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EXECUTIVE  SUMMARY 


Under  the  auspices  of  Total  Quality  Management,  a  small 
group  of  Government  and  industry  specialists  examined  the  existing 
training  device  acceptance  test  process  for  potential  improvements. 
The  agreed  to  mission  of  the  Air  Force/ Industry  partnership  was  to 
continuously  identify  and  promote  implementable  approaches  to 
minimize  the  cost  and  time  required  for  acceptance  testing  while 
ensuring  that  validated  performance  supports  user  training 
requirements.  Application  of  a  Total  Quality  Process  Improvement 
Model  focused  on  the  customers  and  their  requirements,  analyzed  how 
work  was  accomplished,  and  led  to  the  identification  and  elimina¬ 
tion  of  several  non-value  added  components  in  current  test 
practices. 


Diverse  technical  and  management  approaches  were  blended 
into  a  single  improved  process  known  as  Simulator  Test  2000 
(ST  2000) .  ST  2000  integrates  timely,  accurate,  and  streamlined 
test  documentation,  provides  safeguards  for  increased  confidence  in 
contractor  verification  testing,  and  improves  on-time  test 
milestone  performance  via  an  optimum  balance  of  Government/ 
contractor  specification  performance  validation  procedures.  The 
proper  mix  of  Government/contractor  responsibilities  is  highlighted 
with  the  creation  of  Government  System  Performance  and  Development 
Evaluation  (SPADE) .  SPADE  eliminates  Government  repetition  of 
previously  conducted  contractor  tests  and  emphasizes  functional 
operational  checks  to  determine  the  ability  to  satisfy  training 
objectives.  This  customer  oriented  approach  emphasizes  acceptance 
of  documented  contractor  engineering  test  results  Ly  the 
Government. 

Functional  mission  testing  has  been  formulated  to  serve 
as  final  Government  acceptance.  This  on-site  test  philosophy 
serves  to  focus  final  test  evaluations  on  remaining  discrepancies 
and  satisfaction  of  site  related  test  requirements.  By  testing  at 
a  functional  level  in  lieu  of  detailed  testing  constructs,  it 
avoids  repeating  prior  Government  or  contractor  tests. 

The  development  of  training  device  acceptance  test 
improvements  has  proceeded  rapidly  and  to  a  successful  outcome.  To 
significantly  reduce  the  number  of  Government  test  requirements, 
the  joint  Air  Force/Industry  team  has  formulated  27  complimentary 
recommendations  surrounding  the  test  process  rendering  improvements 
estimated  to  save  in  excess  of  40  percent  of  Government  test  time 
without  compromising  test  objectives. 
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1.0 


INTRODUCTION 

When  late  during  the  summer  of  1989  President  Bush 
selected  John  Betti  for  nomination  as  Under  Secretary  of  Defense 
for  Acquisition,  Total  Quality  Management  (TQM)  was  fast  becoming 
more  than  a  household  word.  In  fact,  TQM  was  on  a  course  destined 
to  become  an  intrinsic  management  philosophy  within  the  Department 
of  Defense  (DoD)  .  Betti  was  to  bring  to  the  DoD  from  the  Ford 
Motor  Company  TQM  traditions  to  meet  the  challenge  of  providing 
more  defense  for  the  dollar.  For  the  purposes  of  this  report, 
consider  TQM  as  a  leadership  philosophy  that  creates  a  working 
environment  which  promotes  trust,  teamwork,  and  the  quest  for 
continuous  improvement.  Other  essential  elements  of  TQM  require 
dedication,  conviction,  and  a  willingness  to  bring  about  change,  to 
do  the  right  things  right,  the  first  time,  with  the  ultimate  goal 
being  customer  satisfaction. 

1.1  ACCEPTANCE  TESTING  AS  A  CRITICAL  PROCESS 

At  about  the  same  time  as  Betti's  nomination,  the 
Training  Systems  System  Program  Office  (SPO)  at  Aeronautical 
Systems  Division  (ASD/YW) ,  Wright-Patterson  AFB,  Ohio  was  plowing 
new  and  fertile  ground  with  contractors  from  the  training  system 
industry.  Chartered  in  August  1989,  the  YW/Industry  Total  Quality 
Steering  Group  developed  a  mission  "...dedicated  to  continuous 
process  improvement  in  the  acquisition  of  training  products  and 
services  to  produce  the  best  trained  aircrews  and  maintenance 
personnel  in  the  world".  Initial  membership  represented  a  cross- 
section  from  the  training  systems  development  industry  and  included 
the  following  companies: 

o  CAE-Link  Corporation 
o  ECC  International  Corporation 
o  FlightSafety  Services  Corporation 
o  General  Electric  Company 
o  Hughes  Simulation  Systems  Incorporated 
o  Loral  Defense  Systems  Division 

o  McDonnell  Douglas  Training  Systems  Incorporated 
o  SIMTEC  Inc. 

Membership  from  the  SPO  consisted  of  five  functional  experts 
representing  the  disciplines  of  Engineering,  Logistics, 
Contracting,  and  Program  Management.  The  primary  thrust  of  this 
Government/ industry  forum  was  to  identify  and  provide  recommenda¬ 
tions  for  potential  areas  to  improve  the  training  systems 
acquisition  process  using  the  principles  of  TQM.  In  October  1989, 
the  Steering  Committee  completed  the  identification  and  ranking  of 
six  targets  of  opportunity: 

o  Concurrency 

o  Nonvalue  added  Request  For  Proposal  (RFP)  requirements 
o  Program  Manager  protocols 

o  Prototype  development  in  a  production  environment 
o  Improved  engineering  change  process 
o  Acceptance  testing. 

The  first  improvement  area  selected  for  action  was  judged  on  such 
considerations  as  payback  potential,  direct  impact  on  the  user,  and 
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benefits  of  the  results  to  industry.  The  Steering  Committee  then 
used  the  nominal  group  technique  to  identify  the  opportunity  best 
satisfying  the  selection  criteria.  The  nominal  group  technique  is 
a  refinement  of  brainstorming  which  provides  a  structured  decision 
making  process.  The  results  of  this  analysis  showed  development/ 
acceptance  testing  had  twice  the  improvement  potential  of  any 
runner-up  target  of  opportunity. 

Testing  is  the  primary  means  by  which  a  training  device 
is  evaluated  for  compliance  of  the  design/product  against  required 
characteristics  and  system  performance.  Through  a  process  of 
verification,  validation,  and  authentication,  the  adequacy  of 
performance  characteristics  are  determined  along  with  identifica¬ 
tion  of  deficiencies  in  system  performance.  Acceptance  testing  is 
defined  as  any  and  all  contractor  and  Government  activities 
performed  to  verify  device  conformance  to  specified  system/ 
subsystem  performance  requirements. 

The  test  process  provides  contract  closure  and  allows 
training  initialization.  Yet,  despite  its  importance,  the  test 
process  and  accompanying  test  documentation  has  been  reported  as 
byzantine  at  best.  Many  myths  and  misinformation  abound.  There  is 
widespread  belief,  for  example,  in  the  following: 

o  Acceptance  testing  contributes  to  schedule  delay 

o  Testing  is  rigidly  conducted  in  accordance  with  the 
Test  Matrix 

o  The  Government  must  witness  all  acceptance  tests. 

Training  specialists  in  industry  and  Government  have  long  recog¬ 
nized  a  need  for  improvements  in  military  acceptance  testing  of 
ai  rew  and  maintenance  training  devices.  Over  the  past  years  a 
number  of  papers  have  been  written  on  the  subject.  These  worcs 
include : 

o  A  comparison  of  Simulator  Procurement/Prcgram  Prac¬ 
tices:  Military  vs.  Commercial  -  John  Hussar; 

presented  at  the  5th  Interservice/Industrv  Training 
Systems  Conference  (I/ITSC)  1983 

o  Commercial  Simulator  Acquisition:  A  Three-Way  luide  - 
Arthur  Doty,  Harold  Kottman;  presented  at  the  6th 
I/ITSC  1984 

o  A  Relook  at  Determining  Simulator  Requirements  -  David 
Nelson,  Stephen  Leishman;  presented  it  the  Society  for 
Computer  Simulation  International  1984 

o  Bridging  the  Information  Gap  -  J.  Shaw,  William  Lloyd; 
presented  at  the  11th  I/ITSC  1989. 

Despite  these  efforts,  little  improvement  has  been  made 
in  the  testing  process. 

1.2  the  asd  commanders  perspective 

Culminating  the  selection  of  the  device  testing  process 
were  the  remarks  of  Lieutenant  General  Mike  Loh,  then  the  ASD 
Commander,  as  the  keynote  speaker  at  the  12th  Interservice/Industrv 
Training  Systems  Conference.  In  his  address  to  over  3,000  industry 
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and  Government  training  systems  executives,  Gen  Loh  cited  the  YV// 
Industry  Steering  Group  as  a  model  of  what  should  be  done  wit:, 
every  industry  involved  in  defense  procurement,  "...to  establish  a 
workable  forum  for  frank  and  honest  discussion  of  mutual  concerns". 
He  went  on  to  share  his  belief  in  the  success  of  the  joint 
Government/ industry  team  formed  to  study  the  testing  process  and 
prove  the  worth  of  this  partnership  to  the  skeptics  in  industry  ar.d 
others,  "...this  is  why  I  believe  the  group  will  succeed;  its 
members  own  what  they  are  doing.  They  are  building  a  culture 
within  the  industry  based  on  teamwork  and  trust  ar.d  have  set  their 
sights  on  continuous  improvement.  I  hope  that  it  is  a  harbinaer  :.  1 
a  new  attitude  within  the  defense  procurement  world." 

1.3  THE  CRITICAL  PROCESS  TEAM 

The  Critical  Process  Team  (CPT)  was  chartered  by  t':v. 
Steering  Committee  to  investigate  a  high  level,  cross  organize-- 
ticr.r  I  process  having  a  critical  impact  on  satisfying  the 
customer's  requi rements .  The  team  defined  the  testing  process, 
identified  the  ov'ner  of  the  process,  and  developed  a  system  t: 
recommend  action  on  improving  the  process. 

The  Training  Systems  SPO,  in  response  to  the  Steering 
Committee's  request,  agreed  to  provide  two  members  with  test 
management/engineering  experience  previously  trained  in  TQM 
principlf  s  and  problem  solving  techniques.  The  Steering  Committee 
scLicited  additional  team  members  from  each  of  its  member  compa¬ 
nies.  Industry  representatives  were  asked  to  be  well  versed  ir 
training  device  testing,  have  program  management/project  engineer¬ 
ing  experience,  and  have  excellent  visibility  into  their  company's 
operational  policies.  A  time  commitment  of  up  to  20%  active 
participation  in  the  CPT  was  also  requested.  All  team  members  were 
’-squired  to  attend  a  four  dev  TQM/CPT  training  workshop.  Brief 
b-  irarl  es  of  the  CPT  members  are  included  in  Appendix  A. 

l . r . J  Team  Training 

The  Cumberland  Group,  a  subsidiary  of  Armco  Steel 
ccrlucteJ  an  intensive  four  lay  workshop  to  train  CPT  members  how 
to  analyze  and  improve  the  process.  The  Training  Systems  SPC 
i greed  to  fund  t,.e  training  for  each  CPT  member.  Team  member:, 
gained  i  common  understanding  of  the  CPT  purpose  and  were  able  t;. 
come  to  a  consensus  on  how  the  acceptance  test  process  is  .i 
summation  of  activities  which  must  be  completed  in  the  course 
providin':  a  product  or  service. 

Effective  working  relationships  were  established  and  the 
team  structure  was  created.  Training  provided  the  beginnings  of  a: 
understanding  of  the  Cumberland  Process  Improvement  Model  methodol¬ 
ogy.  Of  particular  significance  was  the  judgement  of  the  team  that 
training  to  define  and  develop  process  improvements  was  essential. 

1.4  THE  PROCESS  IMPROVEMENT  MODEL 

The  Cumberland  Process  Improvement  Model  (Figure  1.4-1) 
consists  of  five  primary  steps  leading  to  the  elimination  of  non¬ 
value  added  components.  The  model  stresses  that  quality  problems 
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CUMBERLAND  PROCESS  IMPROVEMENT  MODEL 


are  very  often  rooted  in  the  process  that  produced  them.  A  change 
in  the  process,  therefore,  is  required  to  achieve  meaning -a, 
improvement  and  not  just  merely  eliminate  the  symptoms.  This  is 
the  foundation  upon  which  the  process  management  approach  to 
quality  improvement  is  built. 

The  objective  of  process  management  is  to  focus  on  the 
customers  of  a  process,  determine  what  the  customer's  requirements 
are,  analyze  how  work  will  be  accomplished,  and  identify  and 
eliminate  the  sources  of  waste  in  the  process. 

Define  Improvement  Opportunity:  The  purpose  of  this  step 
is  to  develop  a  clear  understanding  of  the  team's  tasks.  Expecta¬ 
tions  of  what  was  to  be  accomplished  were  clarified.  Indicat 
(measures)  of  improvement  and  improvement  goals  used  to  guide  ' 
team  as  it  searched  for  areas  of  process  adjustments  were  aar> 
upon . 


Collect  Data:  The  objective  of  this  step  is  to  move  £r:v 
the  statement  of  a  problem  to  a  more  complete  description  of  the 
current  process.  Guidance  was  received  from  the  Training  Systems 
SFO  Program  Director,  as  well  as  the  YW  Steering  Committee. 
Performance  measures  of  several  completed  Government  test  programs 
were  analyzed  to  further  refine  and  finalize  the  process 
indicators. 


Analyze  the  Opportunity:  This  step  is  used  to  first 

identify  areas  of  waste  in  the  current  process  and  the  associated 
root  causes  of  each  waste  area.  Waste  is  defined  as  any  activity 
that  does  not  add  value  to  the  process  and  was  viewed  as  the 
primary  opportunity  for  improving  the  process.  Data  collected  from 
the  previous  step  was  used  to  identify  and  prioritize  waste  and 
foc  us  efforts  on  the  high  payback  areas.  In  the  final  element  -  : 
this  '/or,  the  root  causes  of  each  major  waste  area  v<-r- 
Ldent i t i ed . 

Develop  Solutions;  The  intent  of  his  step  is  to  generate 
uliemalive  ways  to  eliminate  root  causes  of  waste.  The  team 

concentrated  on  ways  to  significantly  change  the  process  instead  : 
merely  making  minor  adjustments.  With  no  assumed  constraints,  the 
team  visualized  the  "perfect"  process  to  fora  an  understanding  c: 
what  could  really  be  achieved,  even  if  in  stages  rather  than  ail 
once . 


Implement  Improvements:  This  final  step  is  designed  tc 
improve  f  he  process  through  a  series  of  recommendations  resulting 
from  solutions  developed  during  the  prior  analysis  step.  Continual 
improvement  is  made  by  planning  the  modification,  engaging  the 
plan,  checking  the  results,  and  making  adjustments  based  on  the 
results . 


For  each  solution  developed,  the  CPT  proposed  a  recommen¬ 
dation  and  agreed  to  advise  those  assigned  to  implement  each 
recommendation . 
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Figure  1.4-2  depicts  the  action  plan  that  identifies  the 
key  events  of  the  process  improvement  model  and  when  they  were  to 
be  completed. 


1-6 


1 


2.0  DEFINITION  OF  THE  IMPROVEMENT  OPPORTUNITY 

The  process  of  developing  a  clear  understanding  of  the 
CPT's  task  and  clarifying  expectations  for  improvements  in 
simulator  testing  was  the  team's  first  major  assignment.  The 
mission  statement  that  follows  was  developed  in  order  to  clearly 
define  the  purpose  and  reason  for  existence  of  the  CPT.  It  was 
formulated  to  indicate  the  partnership  between  the  Air  Force  and 
industry  in  attaining  a  common  goal.  The  mission  statement  was 
coordinated  with  the  owner  and  approved  by  the  steering  committee. 

"We  are  the  YW/Industry  Partnership  CPT, 
committed  to  continuously  identify  and  promote 
implementable  approaches  to  minimize  the  cost 
and  time  required  for  acceptance  testing  of 
aircrew  and  maintenance  training  devices  and 
to  insure  that  these  devices  support  the 
users '  needs . " 

2 . 1  PR0CES8  BOUNDARIES 

The  CPT  clarified  the  scope  of  the  process  to  be  studied 
by  defining  the  boundaries  of  the  testing  function.  There  are  many 
relevant  decisions  pertaining  to  testing  made  long  before  formal 
tests  are  conducted.  Therefore,  the  start  of  the  process  was 
considered  to  be  the  first  time  the  Government  sought  responses 
from  industry;  namely,  the  release  of  the  draft  RFP .  The  evolution 
of  test  requirements  proceeds  through  contract  award,  design 
reviews,  manufacture,  test,  and  site  acceptance  with  the  signing  of 
the  Material  Inspection  Receiving  Report  (DD250) .  The  testing 
process  boundaries  were  then  formally  defined  as  the  release  of  the 
draft  RFP  until  formal  sign-off  of  the  DD250  for  purposes  of 
constructing  the  test  process  flowchart. 

2.2  TEST  PROCESS  FLOWCHART 

The  test  process  flowchart  was  constructed  to  depict  all 
the  acquisition  management  tasks  used  to  develop  and  produce  a 
training  device.  The  process  was  initially  defined  using  the 
flowchart.  Test  activities  were  then  further  analyzed.  The  entire 
flowchart  and  narrative  description  is  provided  in  Appendix  B.  The 
chart  is  separated  with  the  top  showing  those  tasks  performed  by 
the  Air  Force  and  the  bottom  depicting  those  of  industry. 

The  flowchart  depicts  a  typical  training  device  develop¬ 
ment  and  may  vary  somewhat  depending  on  a  particular  program's 
requirements.  Review  of  the  existing  process  revealed  that  there 
were  several  test  repetitions,  five  different  Test  Readiness 
Reviews  (TRRs) ,  and  seventeen  possible  delay  paths. 

2.3  IMPROVEMENT  OBJECTIVES 

While  process  improvement  is  considered  ongoing, 
enduring,  and  not  typically  time-related,  goals  were  considered 
necessary  in  order  to  measure  progress  toward  the  achievement  of 
the  mission  objectives.  These  goals  were  intended  to  promote  the 
concept  that  quality  must  be  enhanced  while  reducing  the  cost  and 
time  required  for  acceptance  testing  of  training  devices.  In  this 
regard,  the  CPT  developed  the  following  improvement  goals: 
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1.  Minimize  the  number  of  test  activities  withe 
compromising  performance  validation 

2.  Foster  timely,  accurate,  streamlined  development 
test  documents 

3.  Facilitate  Government  acceptance  of  qualified  contra 
tor  test  results 

4.  Tailor  application  of  Mil-Spec/Std  test  requireme 
to  emphasize  commercial  practices 

5.  Encourage  improvement  of  on-time  test  milestc 
performance 

6.  Refine  the  test  discrepancy  generation,  tracking 
resolution  process 

7.  Promote  Government/industry  cooperation  and  partne 
ship  for  improving  aircrew/maintenance  training  devi 
testing. 
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3 . 0  DATA  COLLECTION 

3 . 1  PROCESS  INDICATORS 

Process  indicators  are  used  to  measure  the  performance  of 
the  acceptance  test  activities.  Two  approaches  were  used  to 
generate  the  process  indicators.  The  CPT  membership  produced  a 
list  of  parameters  which  measure  the  performance  of  the  acceptance 
test  effort  based  on  specific  testing  experiences.  The  second 
approach  was  to  define  measures  relating  directly  to  the  delay 
loops  in  the  process  flowchart.  Finally,  these  two  lists  were 
evaluated  in  reference  to  the  following  criteria: 

a.  If  the  test  process  is  changed,  will  the  proposed  indica¬ 
tors  be  able  to  measure  the  improvement? 

b.  Can  the  indicator  be  measured  in  real  terms  using  objective 
results? 


c.  Can  data  be  obtained  from  the  companies,  i.e.,  is  the  data 
something  likely  to  be  measured  and  retained  as  part  of  the 
existing  testing  process? 

d.  What  is  the  most  important  data  to  request? 

The  total  list  of  process  indicators  generated  is  shown 
in  Table  3.1-1. 

Using  the  nominal  group  technique,  the  next  st:ep  was  to 
select  from  the  total  list  those  indicators  which  would  be  the 
"best"  measures.  This  was  accomplished  by  each  member  selecting 
the  four  indicators  most  meaningful  to  measure  the  performance  of 
the  acceptance  test  program.  To  downselect,  the  final  indicators 
were  based  on  total  points  (votes)  received  as  noted  in  Table 
3.1-1.  The  CPT  focused  on  the  four  top  indicators  to  be  used  to 
collect  data  for  further  analysis.  These  indicators  are: 

o  Test  milestones  met  or  delayed 
o  Number  of  test  discrepancies 
o  Number  of  days  in  test 
o  Test  documentation. 

3.1.1  Test  Milestones  Met  or  Delayed 

The  first  indicator  is  based  on  schedule  milestones  that 
are  required  to  conduct  a  test  program.  Milestones  were  chosen  to 
be  program/contract  schedule  events  in  anticipation  that  such  data 
would  be  recorded  and  could  be  obtained. 

The  milestones  selected  started  with  TRR  and  ended  with 
the  signing  the  DD250.  The  original  planned  and  actual  dates  were 
recorded  for  each  event  and  period  of  testing.  It  was  to  be 
assumed  that  the  end  of  one  period  of  testing  equaled  the  start  of 
testing  of  the  next  period.  The  list  of  milestones  evaluated 
include: 
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Table  3.1-1.  Process  Indicators 


Score 


DescriDtion 

(Total  Points) 

Rank 

1. 

Meeting  original  contract  RFT  date 

0 

2. 

Hours/Days  required  to  test  in  plant 

0 

3  . 

Number  of  test  documents 

11 

A 
'  t 

4  . 

Number  of  test  interruptions 

0 

5. 

Present  Test  Matrix  which  requires 
analysis/ inspect 

4 

6. 

Less  Government/Industry  testing 
issues 

1 

7  . 

Time  taken  to  test  specific  areas 
vs.  planned  time 

0 

8. 

Number  of  hours  spent  in  retest 

0 

9  . 

Number  of  test  discrepancies 
(categorized) 

17 

2 

10  . 

Number  of  redundant  tests/testing 

6 

11 . 

Number  of  post-test  DRs 

0 

12  . 

Number  of  tests  witnessed  by  the 
Government 

7 

13  . 

Number  of  TDs  unresolved  on  DD250 

0 

14  . 

Reduction  in  residual  problems  post 
DD250 

0 

1-  . 

Number  and  length  of  work  stoppages 
dui.  to  problems  discovered  in 
t  o  s  ting 

0 

16. 

Number  of  days  in  test 

12 

3 

17  . 

Cost  overruns  of  testing 

4 

18. 

Reduction  in  the  test  readiness 
reviews 

0 

19. 

Reduced  hours/days  of  on-site 
testing 

0 
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Table  3.1-1.  Process  Indicators 


Continued 


Score 

Description  (Total  Points)  Hank 


20. 

Time  to  conduct  a  test  procedure 

0 

21. 

Timeliness  of  data  delivered 

0 

22. 

Reduced  test  force  to  perform 
equivalent  testing 

2 

23. 

Number  of  worked  DRs  rejected, 
(Re-submits) 

0 

24  . 

Customer  satisfaction,  (Using 

Command) 

0 

25. 

Reduced  hostility  between 
Government/Industry 

0 

26. 

Numbers  of  steps  in  a  test 
procedure 

0 

27  . 

Size  of  the  Government  test  team 

0 

28. 

Si2e  of  test  documentation 

0 

29. 

Number  of  man-hours  used  in 
redesign 

0 

30. 

Number  of  test  discrepancies 
unresolved 

0 

Ji . 

Number  of  unnecessary  test 
requirements 

5 

32  . 

Number  out  of  scope  DRs 

0 

33. 

Number  of  approval  cycles 

0 

34. 

Test  milestones  met  or  delayed 

18 

35. 

Number  of  days  lost  to 
manufacturing  due  to  testing 

0 

36  . 

Turnaround  time  for  approval 

0 

37  . 

Number  of  ECPs  inserted  in 
original  schedule 

0 

38. 

CDRLs  per  commercial  practices 

3 
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o  Test  Readiness  Review 

o  Computer  Program  System  Generation  (CPSG) 
o  First  Article  Test  Readiness  Review 
o  Development  Test 
o  Operational  Evaluation 
o  Tear  down  and  Pack 
o  Ship 

o  On-Site  Test  Readiness  Review 
o  On-Site  Acceptance  Test 
o  On-Site  Operational  Evaluation 
o  On-site  Sell-off. 

3.1.2  Number  of  Test  Discrepancies 

The  number  of  Test  Discrepancies  (TDs)  generated  dux  ■ 
acceptance  testing  is  a  measure  of  the  training  device  quality,  7 ... 
get  more  insight  into  the  causes  of  TDs,  data  was  requested  r; 
include  total  number  of  TDs,  number  of  TD  re-submits,  number  ci 
post  sell-off  TDs,  and  number  of  TDs  out-of-scope .  Additional  TD 
information  such  as  TDs  against  the  test  procedure,  number  of  7 : 
re-submits,  total  open  at  start  on  TRR,  and  number  of  internal  TD 
during  dry  run  was  not  requested  as  it  was  known  that  only  the 
Training  System  SPO  maintains  this  data. 

3 . 1 . 3  Number  of  Days  in  Test 

The  purpose  of  this  indicator  is  to  isolate  the  delay  t: 
improvement  of  schedule  once  the  unit  is  in  test  by  measuri:.: 
duration.  The  acceptance  test  activity  consists  of  several  phases 
which  are  shown  in  the  process  flowchart.  The  parameters  consid¬ 
ered  important  for  this  indicator  include  planned  days  in  test, 
actual  days  in  test,  number  of  test  hours  per  day,  and  number  ci 
testing  days  per  week. 

Data  was  requested  for  each  phase  of  testing  (duration  i 
days;  plan  vs.  actual).  From  the  results,  the  CPT  selected  three 
phases  to  measure  test  duration  as  a  performance  indicator.  These 
were  in-plant  development  test,  on-site  acceptance,  and  on-site 
operat lonal/FAA  evaluation.  While  other  test  phase  duration  data 
was  available,  the  phases  of  tear-down,  pack,  ship,  install,  and 
on-sit.o  -heckout  were  not  considered  actual  acceptance  testing  an  i 
thus  not  used  in  the  final  analysis. 

3.1.4  Test  Documentation 

The  CPT  membership  considered  test  documentat  i  o  ;• 
excessive.  The  size  of  the  test  procedures,  i.e.,  number  of  pages, 
was  the  means  used  to  measure  this  excess.  In  addition,  the  detail 
to  which  the  procedures  were  written  was  measured  by  the  number  of 
test  steps  per  page. 

The  composition  of  the  Test  Matrix  has  significant 
influence  on  the  size  of  the  test  procedures.  The  data  requested 
for  each  program  was  used  to  identify  the  number  or  percent  of  each 
Test  Matrix  category  of  Inspection,  Analysis,  Demonstration,  and 
Test  used  to  verify  specification  performance.  A  Test  Matrix 
requiring  predominantly  Demonstration  and  Test  causes  lengthy  test 
procedures  to  be  written.  In  addition,  such  procedures  are  usually 
written  at  very  detailed  levels  to  enable  their  use  by  personnel 


3-4 


unfamiliar  with  the  trainer  systems.  Test  Matrix  category  data  was 
not  available  for  commercial  airline  simulators  but  was  retained 
for  miliary  program  analysis. 

To  get  an  objective  measure  of  the  complexity  of  the 
system  under  test,  the  number  of  central  processing  units  and  lines 
of  code  were  used  as  indicators.  In  addition,  the  number  of 
subsystems  within  a  trainer  could  be  used  to  indicate  the  complex¬ 
ity.  With  this  data  it  was  hoped  to  determine  if  a  trend  exists 
between  the  complexity  of  the  trainer  and  the  size  of  the  test 
procedures. 

3 . 2  PROGRAMS  SURVEYED 

After  settling  on  which  indicators  should  be  used,  it  was 
necessary  to  determine  the  possible  sources  of  data  for  the 
information  the  CPT  needed.  In  determining  the  selected  programs, 
the  CPT  focused  on  current/recent  programs  and  the  likelihood  of 
gathering  accurate  data. 

The  following  types  of  trainer  programs  were  included: 
Maintenance  Trainers,  Part  Task  Trainer  (PTT) ,  Mission  Trainers 
(MT) ,  Operational  Flight  Trainer  (OFT) ,  and  Weapon  System  Trainer 
(WST) .  A  brief  description  of  these  trainer  types  is  included  in 
Appendix  C.  The  final  list  of  programs  selected  for  data  collec¬ 
tion  is  as  follows: 

o  F-16  Air  Defense  Fighter  Cockpit  Procedures  Trainer 
(General  Dynamics) 
o  F-15E  WST 
(Loral) 

o  F-16  LANTIRN  PTT 
(ECC) 

o  F-15  LANTIRN  PTT 
(ECC) 

o  GBU-15  PTT 
(Hughes) 

o  AV-8B  Night  Attack  WST 
(McDonnell  Douglas) 
o  C-5B  WST 

(FlightSafety  Services  Corp) 
o  EF-111A  OFT 
(AAI) 

o  F-16  Maintenance  Trainers 
Armament  (Hughes) 

Electrical  (ECC) 

Engine  Operation  (ECC) 

Propulsion  (ECC) 
o  F-15  Maintenance  Trainers 

Integrated  Avionics  System  (Hughes) 

Electrical  Power  and  Lighting  (Hughes) 
o  CH-47D  Electrical  Systems  MX  Trainer 
(Daedalean) 

o  AH-1S  Electrical  Systems  MX  Trainer 
(Daedalean) 
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o 


UH-1N  Basic  MX  Trainer 
(Daedalean) 
o  B-1B  WST 
(Boeing) 

3 . 3  QUESTIONNAIRE 

The  development  of  the  questionnaire  focused  specif ira 
on  the  indicators  used  to  gather  supporting  data.  The  mil  it 
questionnaire  is  shown  in  Appendix  D. 

3 . 4  BENCHMARKING 

Benchmarking  is  the  concept  of  "who  does  it  best", 
approach  was  to  identity  possible  candidates  for  the  CPT 
evaluate  as  "best."  The  ><;--nrKnark  selection  criteria  in-..  : 
str ingent  testing,  use  of  • cmcrcial  practices,  and  exist 
measures  relating  to  tna  :  n  process  indicators.  Can  '::.! 
benchmark  sources  were: 


o 

Airline  Simulator 

Buyers 

o 

o 

Simulation  Indust 
TQM  award  winners 

ry 

o 

o 

Other  TQM  intense 
NASA  Simulators 

companies 

o 

Nuclear  power  p  > 

Simulators 

Searching  for  hi.ghl  efficient  test  methods  tc  bench- 
the  tea-  visited  the  TQM  landmark  companies,  Motorola  and  1 
Unfortunately,  these  firms  did  not  yield  the  anticipated  benchma 
as  their  processes  were  invar; ibly  on  too  small  a  scale  to  giv 
comparable  measurement.  Typic  •  t  Motorola  acceptance  tests  ran  1 
than  twenty-five  minutes.  This  provided  an  insufficient  compari 
when  measured  with  the  complexity  of  a  modern  simulator.  IBM  to 
in  a  similar  manner  to  the  simulator  industry,  but  again 
relut'v-  -  y  small  scale  o  t;  devices  made  comparison  diffi.u 
In  ids  i *  .  n ,  IBM  did  not  '.  :  .  rent  data  they  wished  to  re' "a 

Otner  companies  investigated  by  the  CPT  membership 
where  a  questionnaire  was  sent  are  as  follows: 

o  Walt  Disney  Irons ineering 
o  Loral  Defense  Systems  Division 
o  CAE  Electronics.  LTD 
o  Link  Tactical  s  . mu 1 at ion  Division 
o  American  Airlines 
o  United  Airlines  Training  Center 
o  Motorola 
o  IBM  Owego 
o  Xerox  Corporation 
o  Raymond  Corporation 
o  Northwest  Airlines 

o  Martin-Marietta  Missile  Systems  Division 
o  NASA 

o  Hughes  Satellite  Division 
o  Westinghouse  Inc. 
o  Boeing  Commercial  Airplane  Co. 
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o  Delta  Airlines 

o  US  Air. 

The  replies  to  the  CPT  membership  inquiries  and  question¬ 
naire  were  extremely  poor.  Successive  follow-ups  by  team  members 
did  little  to  elicit  further  responses.  Many  indicated  they  felt 
their  testing  approach  was  sufficiently  different  to  make  it 
unsuitable  for  our  purposes.  It  rapidly  became  apparent  that 
integration  and  test  of  a  full  flight  simulator  is  a  uniquely 
challenging  task  not  commonly  encountered  in  other  industries. 

At  that  point  the  team  decided  to  focus  on  the  commercial 
airline  simulator  industry  as  the  candidate  for  "best".  This  was 
based  on  the  fact  that  they  buy/builu  a  product  very  similar  to  the 
Air  Force,  use  commercial  standards,  and  must  pass  stringent 
acceptance  testing  conducted  for/by  the  FAA .  Seven  commercial 
devices  provided  adequate  data  for  benchmarking.  All  were  true 
commercial  devices  with  commercial  customers,  except  the  C-.5B  ATS, 
where  United  Airlines,  using  best  commercial  practices,  purchased 
C-5B  WSTs  from  CAE  for  that  USAF  training  program.  All  commercial 
devices  studied  were  at  least  FAA  Phase  rl* certified  high  fidelity 
training  devices  and  equivalent  to  OFT/WST  systems  but  without 
tactics  simulation.  Lata  was  collected  from  United,  Northwest, 
Delta,  US  Air,  and  the  Boeing  Company  and  used  for  subsequent  data 
analysis.  The  questionnaire  used  tc  collect  data  from  the  buyers 
of  commercial  aircraft  simulators  is  provided  in  Appendix  E. 

3 . 5  DATA  SUMMARY 

The  results  of  the  questionnaire  were  captured  in  the 
Data  Tables  (Appendix  F)  .  The  raw  data  (pages  1  through  C)  was 
analyzed  and  incorporated  into  the  final  summary  sheet  (pages  9  and 
10) .  Several  very  important  adjustments  were  made  to  produce  the 
summary  sheet.  The  first  adjustment  was  to  eliminate  companies, 
programs  that  did  not  respond  to  the  questionnaire  and  those  who 
felt  ■‘■.heir  process  was  different.  In  addition,  programs  were 
dropped  if  sufficient  data  was  not  available.  Only  limited  data 
was  available  for  maintenance  trainer  programs  due  to  less 
stringent  test  requirements.  Comparison  data  for  process  indica¬ 
tors  was  therefore  non-existent.  The  B-1B  WST  was  examined  and  the 
data  ultimately  discarded  due  to  this  program's  unique  difficulties 
and  numerous  setbacks  which  made  it  a  totally  atypical  measurement 
point.  As  a  result,  six  military  programs  and  the  seven  benchmark 
commercial  programs  remained  for  further  evaluation  as  contained  in 
the  final  summary  sheets.  The  number  of  test  milestones  was 
reduced  from  those  identified  in  the  questionnaire  to  get  a  one-to- 
one  mapping  of  significant  milestones  between  the  military  and 
commercial  programs. 
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4.0  ATALYSIS  OF  IMPROVEMENT  OPPORTUNITIES 

There  is  a  fine  distinction  between  a  problem  and  an 
opportunity.  In  this  phase  of  the  CPT  effort,  as  problems  were 
substantiated,  opportunities  became  apparent.  The  serious  issue 
then  was  to  focus/ select  opportunities  that  satisfied  the  mission 
statement . 


The  raw  data,  after  being  reviewed  fo:  omissions,  was 
organized  for  the  purpose  of  identifying  waste  areas  in  the  test 
process  and  subsequently  determining  thair  root  causes.  The  data 
was  grouped  according  to  the  four  process  indicators  and  studied 
for  information  and/or  conclusions  that  could  be  drawn  from  the 
data  sets.  This  effort  resulted  in  a  series  of  provocative 
questions  recorded  in  Appendix  G  and  eight  histograms  that  led  to 
the  identification  of  major  waste  are^s. 

Concurrently,  the  detailed  process  flowchart  was  refined, 
thus  creating  a  clear  picture  of  the  current  military  simulator 
Lest  process.  When  evaluated  against  the  datasets,  additional 
waste  areas  were  defined.  Each  waste  area  was  then  decomposed  to 
identify  its  root  causes. 

4 . 1  CHARTS 

The  data  was  plotteu  to  obtain  a  visual  representation  to 
assist  in  the  analysis  effort.  Programs  were  arranged  in  order  of 
complexity  as  j  ldged  ov  the  CPT.  The  analysis  consisted  of 
studying  each  chart  tc  identify  relationships,  trends,  and 
observations. 

4.1.1  Test  Program  Overview 

Chart  1  and  Chart  2  depict  milestones  met  or  delayed  for 
military  and  commercial  programs.  (Figures  4. 1.1-1  and  4. 1.1-2) 

Observations 

o  Military  first  article  systems  are  typically  late 
start ina  test 

o  Commercial  systems  are  usually  on  time  starting  test 

o  Once  test  starts,  it's  usually  performed  according  to 
the  planned  schedule  or  the  magnitude  if  delay  is  held 
constant 

o  Planned  tests  are  realistic  in  terms  of  schedule. 
However,  test  often  starts  before  the  device  is 
completely  ready 

o  Some  programs  recover  schedule  during  testing 

o  The  maximum  tes;  schedule  adjustment  is  five  to  ten 
percent  of  that  planned 

o  Commercial  testing  appears  to  finish  on  schedule. 

4.1.2  Test  Discrepancies 

Chart  3  depicts  various  categories  of  discrepancies  for 
both  military  and  commercial  programs.  (Figure  4. 1.2-1) 
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Observations 


o  Re-submits  do  not  affect  schedule 

o  Approximately  ninety  percent  of  the  TDs  are  resolved 
during  the  testing  period 

o  The  customer  accepts  the  trainer  with  approximately  10 
percent  of  the  TDs  remaining 
o  Out-of-scope  TDs  do  not  affect  the  test  schedule 
o  No  correlation  exists  between  start  of  test  and  the  TD 
count 

o  The  programs  researched  had  an  average  of  approxi¬ 
mately  600  discrepancies 

o  Total  TD  count  appears  to  correlate  with  device 
complexity. 

4.1.3  Test  Days  vs.  TDs 

Chart  4  depicts  the  number  of  days  in  test  compared  to 
the  quantity  of  test  discrepancies.  (Figure  4. 1.3-1) 

Observations 

o  Quantity  of  TDs  appear  to  correlate  to  days  in  test 
(duiation) . 

4.1.4  Test  Phase  Duration 

Charts  5  and  6  depict  planned  and  actual  days  for 
selected  test  program  activities  for  military  and  commercial 
programs.  (Figures  4. 1.4-1  and  4. 1.4-2) 

Observations 

o  The  ratio  of  military  to  commercial  test  phase 
durations  correlates  and  supports  the  fact  that  the 
military  schedule  is  realistic, 
o  Operational/FAA  evaluations  meet  planned  duration 
o  Typically,  both  military  and  commercial  programs  test 
ten  to  twelve  hours  per  day 

o  Planned  test  days  correlate  well  with  the  actual  test 
days 

o  Breaks  in  the  test  schedule  for  redesign  activity  do 
not  seem  to  significantly  affect  days  in  test 
o  Complexity  does  not  appear  to  affect  the  planned 
versus  actual  test  schedule. 

4.1.5  Pages  of  Test  Procedures 

Chart  7  depicts  the  size  of  test  procedures.  (Figure 

4. 1.5-1) 


Observations 

o  No  correlation  exists  between  the  number  of  pages  and 
test  duration 

o  Allowing  for  a  delta  between  an  FAA  Phase  II  device 
for  military  tactics  packages  found  in  wsTs,  the 
relative  size  of  military  and  commercial  procedures 
are  the  same 
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PAGES  OF  TEST  PROCEDURES 
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o  Size  of  the  test  procedures  correlates  well  w;‘- 
trainer  complexity. 

4.1.6  Test  Matrix  Factors 

Chart  8  depicts  a  comparison  of  various  test  categoric: 
for  military  programs.  (Figure  4. 1.6-1) 

o  Data  indicates  a  low  utilization  of  analysis  and  a 
high  utilization  of  test  and  demonstration  to  veriiy 
specification  performance 

o  No  common  application  exists  from  program  to  program 
governing  the  use  of  inspection,  analysis,  demonstra¬ 
tion,  and  test. 

4.2  IDENTIFICATION  OF  WASTE  AREAS 

Each  chart  was  individually  reviewed  to  identify  possible 
waste  areas  in  the  test  process.  Graphical  analysis  assisted  ir. 
producing  a  better  definition  of  the  problems.  The  team  used 
tabular  data,  histograms,  the  process  macro  flow,  and  comments  on 
the  questionnaires  to  assist  and  confirm  proper  identification  of 
critical  process  indicators  and  waste  areas.  Eight  waste  areas 
were  identified  for  subsequent  root  cause  analysis. 

WASTE  AREAS 

1.  Delay  in  start  of  test 

2.  Redundant  testing 

3.  Detailed  customer  subsystem  performance  verification 

4.  Test  Discrepancies 

5.  Excessive  test  Documentation 

6.  Test  Interruptions 

7.  Multiple  test  readiness  reviews 

8.  Computer  Program  System  Generation  (Cold  Start). 
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5.0  DEVELOPMENT  OF  SOLUTIONS 

The  solutions  described  are  based  on  eleven  months  of 
intensive  study,  data  collection,  and  analysis.  With  the  knowledge 
obtained  to  this  point,  the  current  test  process  flowchart  was 
revisited.  By  removing  all  constraints,  bias,  and  myths,  then 
applying  insight  gained  from  the  data,  an  idealized  flowchart  was 
generated  to  visualize  a  test  process  void  of  identified  wastes. 

This  flowchart,  in  conjunction  with  the  information  gained  from 
identifying  root  causes,  provided  the  basis  for  developing 
solutions . 


Although  the  solutions  are  specific  in  nature,  they  are 
not  intended  to  be  perceived  as  the  only  solution  but  rather  as  the 
CPT's  recommendations  based  on  the  research  conducted.  In  a  few 
cases  the  solution  for  the  root  causes  of  the  waste  areas  identi¬ 
fied  may  require  further  refinement.  For  the  majority,  there  are 
solutions  formulated  that  are  easily  and  readily  implementable .  It 
should  be  noted  that  all  solutions  had  a  consensus  of  the  CPT 
membership. 

5.1  WASTE  AREA  #1  -  DELAY  IN  START  OF  TEST 

5.1.1  Cause  Summary 

The  following  factors  were  identified  by  this  CPT  as 
being  directly  related  to  test  delays  (Figure  5- 1.1-1): 

o  Late  Government  identification  of  minimum  training 

needs/ funding 

o  Poorly  defined  requirements 
o  Incomplete  design 

-  Requirements  not  complete 

-  Data  not  available 

-  Resources  not  available 

-  Inefficient  implementation  of  new  technology 
o  Manufacturing  not  complete 

-  Government  Furnished  Equipment.  (GFE) /Contractor 
Furnished  Equipment  (CFE)  not  available 

-  Inadequate  subcontractor/vendor  management 

o  Hardwaro/Sof tware  Integration  in-process  measurement 

criteria  lacking. 

5.12  Solutions 

Program  delays  due  to  late  Government  identification  of 
training  needs  and  inadequate  program  funding  often  cause  the 
program  planning  phase  of  the  procurement  process  to  be  incomplete. 
Inadequate  research  during  this  period  results  in  poorly  defined 
requirements.  This  often  results  in  the  requirements  "creep" 
phenomenon  during  later  stages  of  design  development.  The  problem 
is  further  compounded  when  the  contractor  accepts  these  nebulous 
requirements  and  consequently  fails  to  perform  to  Government 
expectations.  Thorough  completion  of  the  Training  Systems 
Requirements  Analysis  prior  to  the  release  of  the  RFP  will 
greatly  assist  in  well-defined,  realistic  requirements  to  provide 
a  sound  basis  for  contractor  scheduling,  pricing,  and  technical 
performance. 
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Waste  Area  #1 


Resources 

New 

Technology 


Problems  associated  with  the  contractor's  failure  to 
complete  the  device  design  prior  to  testing  due  to  incomplete  data 
may  be  decreased  or  eliminated  oy  identifying  missing  data  early 
during  the  program  and  establishing  join*.  contractor/Government 
interpretation.  This  interpretation  should  then  be  formally 
included  in  the  Design  Criteria  List  (DCL) .  Early  involvement  of 
Subject  Matter  Experts  {SMEs)  will  also  help  alleviate  problems 
associated  with  a  lack  of  data  by  providing  an  "on-line”  data 
source  during  the  design  development  phase.  In  addition,  implemen¬ 
tation  of  the  data  generation  and  management  principles  identified 
in  the  Simulator  Data  Integrity  Program  sponsored  by  the  Training 
System  SPO  under  contract  number  AF33657-88-C-2168  should  be 
considered  to  ensure  accurate  and  complete  data  is  provided  by  the 
prime  weapons  system  contractor.  The  Simulator  Data  Integrity 
Program  serves  as  a  source  data  process  standard  and  provides  the 
following: 

o  A  tailorable  working  tool  clearly  defining  the 
management  and  technical  processes  to  be  applied 
during  the  acquisition  and  implementation  of  source 
data 

o  Processes  providing  for  quality  and  uniformity  among 
the  multiple  suppliers  of  data  whicn  support  a  single 
training  system 

o  A  standard  that  yields  source  data  products  with 
excellent  integrity,  quality,  and  supportabil ity 
characteristics . 

The  problem  of  unavailable  resources  centers  around 
delays  caused  by  events  leading  up  to  and  including  Hardware/ 
Software  Integration  which  have  been  determined  to  be  especially 
significant  by  this  CPT.  For  example: 

o  Hardware/Software  Integration  suffers  from  poor 
planning  and  implementation 

o  Ability  to  manage  the  Hardware/Software  Integration 
process  has  been  lacking 

o  Start/stop  criteria  and  in-process  measurement  tools 
have  been  non-existent. 

The  training  system  industry  is  encouraged  to  investigate/solve 
problems  in  these  areas.  It  is  believed  that  with  the  correction 
of  these  problems  the  resources  availability  issues  will  be 
similarly  resolved. 

Schedule  risks  associated  with  utilizing  new  technology 
in  the  device  design  can  be  mitigated  by  developing  prototype 
testing  procedures  to  mature  the  technology  prior  to  Hardware/ 
Software  Integration.  These  procedures  can  be  reduced  on 
subsequent  production  quantities  as  the  risk  of  the  technology 
decreases. 

GFE  availability  problems  can  be  reduced  by  implementing 
a  system  in  which  the  Air  Force  procures  the  required  training 
system  components  from  the  prime  weapons  system  contractor  as  soon 
as  GFE  requirements  are  known.  An  alternative  is  to  have  the 


Government  include  in  the  weapons  system  contract  the  requirement 
to  enter  into  an  associate  contractor  agreement  with  the  simulate: 
contractor  to  supply  the  necessary  components.  Alternatives  tc 
using  GFE  should  be  explored  such  as  the  use  of  "Olive  Drab" 
commercial  components  which  are  essentially  equivalent  to  MIL-SPEC 
hardwa'’-  items  that  have  n  been  sponsored  or  designed  by  the 
Government.  The  use  of  equivalent  non-military  components  is  yet 
another  option. 

Inadequate  subcontract: a r/vendor  management  problems 
not  it  the  Air  Force  direct  line  of  responsibility;  however,  th : 
Government  can  influence  the  prime  contractor  to  address  this  area 
to  redere  the  risk  to  testing.  Suggestions  for  industry  improve¬ 
ments  i-’tlude: 

o  Avoid  multi-levei,  multi-party  subcontractor  arrange¬ 
ments  on  majoi  device  components  where  the  act: 
supp] .er  has  nr  direct  link  to  the  prime 
o  Establish  a  strong  subcontractor/ vendor  management 
team  to  take  responsibility  for  the  supplier  r 
performance 

o  Use  on-site  representatives  when  necessary  to  close iy 
monitor  supplier  performance 
o  Use  Material  Requirements  Planning  (MRP)  packer f-s 
help  schedule  vender  delivery 
o  Develop  reliable  second  sources  ‘or  high  r. 
components 

o  Insist  on  monitoring  and  reviewing  major  subcontractor 
performance  on  a  regularly  scheduled  basis  to  identify 
potential  problem  areas. 

5.2  WASTE  AREA  #2  -  REDUNDANT  TESTING 

5 . 2 . 1  Cause  Summary 

The  following  factors  have  been  identified  as  contrite", 

inq  to  the  problem  oi  redundant  testing  (Figure  5. 2. 1-1): 

o  No  Government  recourse  after  buy-off 
o  Improper  engineering  test  procedures 

-  Engineering  procedures  not  repeatable 

-  Results  not  documented. 

5.2.2  Solutions 

The  customer  views  acceptance  testing  as  his  "one  and 
only  shot"  at  discovering  all  system  problems.  This  results  m 
aggressive,  excessive  testing  by  the  Government  to  ensure  the 
continued  performance  of  the  simulator  throughout  the  required  life 
cycle.  The  contractors  can  instill  confidence  in  their  product  and 
thus  lessen  the  need  for  extensive  testing  by  providing  a  more 
comprehensive  performance  warranty  package  similar  ir  scope  to 
those  currently  available  to  commercial  airlines. 

Redundant  testing  often  occurs  as  a  result  of  poor 
contractor  testing  procedures.  The  Government  does  not  accept 
Contractor  Verification  Testing  (CVT)  results  as  valid  and  usually 
re-runs  the  tests  in  total.  Confidence  in  contractor  test  results 
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Figure  5.2. 1-1.  Waste  Area  #2 
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can  be  established  by  increasing  the  quality  of  in-plant  testing 
and  including  advisory  SME  involvement  during  CVT.  This  solution 
forces  the  contractor  to  conduct  in-plant  tests  which  are  repeat- 
able  and  wel 1 -documented .  Consistent  contractor  Le&L  results  will 
increase  the  likelihood  of  Government  acceptance  of  the  data 
generated  and  eliminate  the  need  for  repeating  previous  contractor 
testing.  Failure  to  properly  perform  and  document  CVT  results  in 
jeopardized  on-site  device  performance  and  reduced  profit  due  to 
the  attending  schedule  delays  and  additional  contractor  resource;: 
necessary  to  upgrade  the  device  to  an  acceptable  condition.  Witt 
reduced  Government  in-plant  testing,  thorough  functional  missis,: 
tests  to  be  conducted  by  the  Government  prior  to  acceptance  on-site 
will  become  a  necessity. 

5.3  WASTE  AREA  #3  -  DETAILED  CUSTOMER  SUBSYSTEM  PERF ORMAIL.'E 

VERIFICATION 

5.3.1  Cause  Summary 

The  CPT  has  identified  the  following  factors  as  ccrvcr : 
uting  to  overly  detailed  perf o-mance  verification  testing  (Ficru: . 
5. 3. 1-1): 

o  Contractor  test  results  not  available/documented 
o  Traditional,  bottons-up  test  techniques 
o  Performance  risks  associated  with  now  technology . 

5 . 3 . 2  Solutions 

The  need  for  detailed  Government  performance  verification 
to  the  subsystem  level  can  t-  liminated  by  instituting  improves 
contractor  test  procedures.  T.  rototype  tests  should  be  developed 
for  high  risk,  new  technologies  prior  to  Hardware/Software 
Integration  until  a  satisfactory  confidence  level  is  reached.  This 
compliments  the  suggestion  to  mature  new  technology  contained 
5.1.2.  Traditional  betters.-- -up  testing  should  no  longer  i  - 
performed  by  the  Government  *r  j  will  require  a  modification  of  t:; 
Test  Matrix  for  inclusion  the  MIL-PRIME  Air  Force  Guidance- 

specifications  87241  for  flight  simulators  and  87228  for  mainte¬ 
nance  training  devices.  Instead,  these  procedures  should  b*> 
completed  during  CVT.  The  procedures  and  test  results  should  i 
thoroughly  documented  for  Government  review,  thus  allowing  one  t.ir 
cost  effective  and  efficient  system  level  acceptance  testing. 

5.4  WASTE  AREA  #4  -  TEST  DEFICIENCIES 

5.4.1  cause  Summary 

The  CPT  has  identified  the  following  problem  areas  which 
contribute  to  TDs  (Figure  5. 4.  1-1): 

o  Lack  of  trained  resources 
o  Invalid  test  procedures 
o  Poorly  defined  operational  performance 
o  Data  shortfalls 
o  Incomplete  contractor  testing. 
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Figure  5.4. 1-1.  Waste  Area  #4 
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5.4.2  solutions 

TDs  are  often  a  result  of  poor  quality  in  the  areas  of: 

o  Workmanship 
o  Configuration  Management 
o  Software  Design 
o  Hardware  Design 
o  Corrective  Action. 

Proper  training  of  test  personnel  in  these  areas  is 
essential  in  limiting  the  number  of  TDs  written  against  a  given 
device.  Proper  training  ensures  that  test  procedures  are  properly 
generated  and  performed  and  test  personnel  are  able  to  sufficiently 
measure  device  performance  against  performance  criteria.  Imple¬ 
menting  in-house  training  programs  based  on  the  principles  of  TQM 
to  create  a  climate  of  pride,  teamwork,  and  ownership  has  great 
potential  to  alleviate  excessive  TDs  and  increase  overall  quality. 

Poorly  defined  performance  requirements  also  contribute 
to  unnecessary  TDs.  This  problem  relates  back  to  a  failure  of  the 
Government  to  identify  training  needs  early  in  the  program  and 
complete  the  Training  Requirements  Analysis  Report  and  Operator  and 
Maintenance  Analysis  Report  as  discussed  in  section  5.1.2,  Waste 
Area  1.  Once  identified,  the  contractor  must  interpret  test 
requirements  correctly  and  obtain  Government  concurrence. 

A  closely  related  problem  is  a  lack  of  performance  data. 
Data  shortages  can  be  alleviated  by  making  SMEs  available  through¬ 
out  the  design,  development,  and  CVT  testing  phases  of  the  program. 
SME  involvement  in  design  reviews  is  especially  encouraged  to 
clarify  design  data  assumptions  and  resolve  ambiguities  with  the 
results  then  formally  documented.  Also,  implementing  the  Simulator 
Data  Integrity  Study  recommendations  will  help  ensure  effective 
data  flow  between  the  prime  airframe/avionics  developer  and 
training  system  manufacturers  as  previously  outlined  in  Waste  Area 
#1. 


|  The  final  area  identified  as  contributing  to  TDs  is 

testing  prior  to  satisfactory  completion  of  CVT.  The  many  factors 
involved  in  this  problem  and  the  CPT's  recommended  solutions  have 
been  previously  discussed  in  Waste  Area  #1. 

It  should  be  noted  that  TDs  are  symptoms,  not  causes. 
Root  causes  which  give  rise  to  TDs  have  been  identified  and 
(  solutions  discussed  in  each  of  the  other  waste  areas. 

5.5  WASTE  AREA  #5  -  EXCESSIVE  TEST  DOCUMENTATION 

5.5.1  Hl1T"™arv 

The  following  factors  contribute  to  the  problem  of 
excessive  documentation  (Figure  5. 5. 1-1): 

o  Documentation  is  overly  complex  and  detailed  providing 
*  for 

-  Repeatability 

I  -  Skill  level 
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□  Root  Cause 


Figure  5.5. 1-1.  Waste  Area  #5 
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contract 


-  Support  considerations 

-  Test  Matrix  requirements 
o  Documentation  is  not  coordinated  across 

requirements 

-  Micro  management. 

5.5.2  Solutions 

Overly  complex  and  detailed  documentation  can  be 
alleviated  by  writing  engineering  tests  at  the  functional  level . 
CDRL  items  should  be  changed  to  allow  submittal  in  contractor's 
format.  Greater  emphasis  should  be  placed  in  the  SOW  requirements 
for  contractor  development  of  automated  test  procedures  for  such 
areas  as  aero  and  performance  tests,  initial  conditions,  avionics, 
nav-aids,  diagnostics,  etc.  A  further  solution  is  to  have 
Government  testing  at  the  mission  level  which  eliminates  the  need 
for  step  by  step  procedures.  Support  considerations  no  longer 
dictate  excessive  test  documentation  as  Contractor  Logistics 
Support  has  been  implemented  for  all  programs.  Another  solution  is 
to  examine  the  Test  Matrix  at  design  reviews  to  minimize  the 
requirements  for  tests  and  demonstrations  based  on  systems  design. 

Documentation  which  is  uncoordinated  across  contract 
requirements  can  be  avoided  by  centralizing  the  responsibility  and 
authority  for  data  requirements  with  the  Government  lead  engineer 
to  eliminate  duplication.  In  addition,  pre-approved  plans  for 
Systems  Engineering  Management,  Systems  Test,  Quality  Assurance, 
Configuration  Management,  Software  Development,  etc.,  should  be 
used  wherever  possible  to  eliminate  the  need  for  separate  plans  for 
each  program. 

5.6  WASTE  AREA  #6  -  TEST  INTERRUPTIONS 

5.6.1  Cause  Summary 

The  following  factors  were  identified  as  causing  schedule 
interruptions  (Figure  5. 6. 1-1): 

o  GFE/CFE  spares  not  available 
o  Schedule  pressure 

-  Acceptable  risk 

o  Poor  systems  analysis/solutions  unsatisfactory 
o  Customer  facility  not  ready 

Lack  of  control  of  the  construction  program 

Lack  of  contractor  design 

o  Software  update  process  errors. 

5.6.2  Solutions 

Test  interruptions  due  to  schedule  pressure  often  result 
from  allowing  known  problems  to  exist  unresolved.  The  contractor 
should  monitor  these  problems  through  the  use  of  established  risk 
management  procedures,  resolve  TDs  as  quickly  as  possible,  and 
ensure  trained  personnel  are  available  in  each  specific  program 
area.  The  detennination  of  acceptable  risk  to  enter  into  Govern¬ 
ment  testing  with  known  problems  occurs  during  TRR.  A  more 
comprehensive  analysis  effort  should  be  made  at  this  time  prior  to 
entering  test  to  avoid  delays,  disruptions,  and  waste. 
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I  I  Root  Cause 


Figure  5.6. 1-1. 


Waste  Area  #6 
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A  lack  of  GFE/CFE  spares  is  often  responsible  for  test 
interruptions.  The  Government  can  solve  this  problem  by  consider¬ 
ing  spares  requirements  when  ordering  components  from  the  weapons 
system  manufacturer.  For  CFE,  the  contractor  should  establish  a 
repair  pipeline  with  the  original  equipment  manufacturer  and  have 
the  system  in  place  prior  to  testing.  Additionally,  the  solution 
for  early  acquisition  of  GFE  by  the  Government  should  be  imple¬ 
mented  as  discussed  previously  in  paragraph  5.1.2. 

Poor  system  analysis  is  the  root  cause  of  many  test 
interruptions  as  unresolved  critical  TDs  often  result.  An  ad  hoc 
team  should  be  established  to  resolve  "show  stopper"  TDs  utilizing 
both  contractor  and  Air  Force  resources  as  required. 

Automated  error  detection  methods  should  be  explored  to 
reduce  interruptions  due  to  software  update  process  errors. 

Modification  or  new  construction  of  a  facility  is  most 
often  accomplished  via  the  Military  Construction  route  (3300 
appropriation  funding).  However,  the  ability  to  use  RDT&E  (3600 
appropriation  funding)  should  be  exploited  where  new  facility 
construction  is  a  requirement.  AFR  80-22  states  that  RDT&E  funds 
may  be  used  to  acquire  industrial  and  RDT&E  facilities  needed  by 
contractors  to  fulfill  R&D  contracts  as  authorized  by  10  USC  2353. 
This  has  been  interpreted  to  mean  that  where  a  facility  is  needed 
by  a  contractor  in  order  to  perform  tasks  required  by  a  R&D 
contract,  that  facility  may  be  provided  through  this  funding. 

5.7  WASTE  AREA  #7  -  MULTIPLE  TEST  READINESS  REVIEWS 

5.7.1  Cause  Summary 

The  following  item  has  been  identified  as  causing 
multiple  TRRs  (Figure  5. 7. 1-1): 

o  Failure  of  the  contractor  to  be  ready  for  test. 

5.7.2  Solution 

Multiple  TRRs  are  costly  to  both  the  Government  and 
contractor,  contribute  to  the  length  of  the  test  schedule,  and 
constitute  a  non-productive  expenditure  of  test  team  resources. 
Data  indicates  that  the  contractor  historically  is  not  ready  at 
TRR.  Multiple  Government  TRRs  can  be  avoided  by  placing  the  burden 
of  test  readiness  solely  on  the  contractor.  The  need  for  multiple 
TRRs  should  be  re-evaluated  and  contract  requirements  written  to 
reduce  the  number  of  TRRs  accordingly. 

5.8  WASTE  AREA  #8  -  C0MP0TER  PROGRAM  SYSTEM  GENERATION 

5.8.1  Cause  Summary 

The  following  factor  has  been  identified  as  contributing 

to  CPSG  requirements  (Figure  5. 8. 1-1): 

o  Lack  of  or  weak  contractor  software  tools 

o  The  need  to  accommodate  changes. 
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□  Root  Cause 


Figure  5.7. 1-1.  Waste  Area  #7 
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□  Root  Cause 


Figure  5. 8.1-1.  Waste  Area  #8 
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5.8.2  Solution 

CPSG  requirements  constitute  a  lengthy  and  unnecessary 
waste  of  resources  when  large  scale,  complex  devices  are  involved. 
Software  tools  and  processes  are  available  which  provide  the  same 
level  of  confidence  in  the  integrity  of  the  software  as  CPSG.  This 
is  particularly  true  for  those  programs  requiring  Ada  software 
language.  Government  certification  of  contractor  software  tools 
and  processes  for  software  configuration  management  and  the 
capability  to  support  changes  should  be  accomplished  prior  to  test. 
The  procedures  for  certification  should  be  included  in  the 
Configuration  Management  Plan.  Certification  would  then  allow  lor 
the  elimination  of  CPSG  requirements. 
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6.0  IMPROVEMENT  RECOMMENDATIONS 

The  CPT  is  recommending  fundamental  changes  to  the 
acceptance  test  process  and  supporting  activities  of  that  process. 
These  recommendations  are  based  on  analysis  of  the  current  test 
methodolgy  and  solutions  formulated  by  the  team  to  eliminate  the 
root  causes  for  identified  wastes.  Accountability  will  be  improved 
by  better  aligning  authority  with  responsibility  for  both  the 
Government  and  contractor  development  test  teams.  Unnecessary  and 
redundant  testing  will  be  eliminated,  test  documentation  will  be 
minimized,  and  the  level  and  type  of  testing  will  be  "iore  focused 
on  satisfying  training  requirements.  In  addition,  a  comprehensive 
and  more  effective  test  assessment  will  be  realized  without 
extensive  TRRs.  Test  procedures  will  be  elevated  to  the  functional 
level  and  Air  Force  SMEs  will  be  made  available  to  assist  the 
contractor  during  the  development  phase.  Test  functionality  will 
not  be  reduced  nor  will  test  integrity  be  compromised  as  a  result 
of  these  recommendations. 

The  CPT  concentrated  on  the  task  of  improving  the 
efficiency  of  acceptance  testing.  However,  "testing"  encompasses 
and  is  influenced  by  a  much  broader  span  of  activity  outside  the 
formal  test  program.  Program  development  tasks  prior  to  acceptance 
testing  were  suspected  by  the  CPT  membership  of  masking  problems 
subsequently  appearing  during  or  delaying  the  start  of  acceptance 
testing.  The  data  collected  supports  prior  suspicions.  CPT 
findings  show  a  major  cause  for  delay  in  fielding  acceptable 
training  devices  is  due  to  activity  that  precedes  the  start  of 
acceptance  testing.  In  particular,  recommendations  made  in  the 
areas  of  design  data,  aircraft  components,  and  Hardware/Software 
Integration  are  emphasized  because  of  their  known  historical  impact 
on  the  test  program.  Correction  of  these  problems  w  11  largely 
avoid  significant  delays  experienced  on  past  programs.  Further,  in 
the  era  of  software  driven  glass  cockpits,  these  problems  will 
assume  even  larger  proportions  if  not  contained. 

The  CPT,  through  the  efforts  of  identifying  areas  of 
waste,  has  formed  additional  recommendations  viewed  as  critical. 
Three  general  recommendations  are  made  to  improve  Government  and 
contractor  test  program  efficiency  through  contract  provisions 
relating  to  contractor  performance  and  are  strongly  recommended. 

To  determine  ideas  for  further  improvements,  a  letter  was  sent  by 
the  Training  Systems  SPO  Program  Director  to  canvas  the  training 
systems  industry.  The  correlation  of  the  comments  with  the 
team's  recommendations  was  excellent.  Virtually  all  companies 
proposed  ideas  similar  to  those  advanced  by  the  CPT.  This  can  be 
interpreted  as  a  very  high  order  of  confirmation  of  the  team's 
analysis.  The  supporting  inputs  received  from  industry,  the 
success  on  programs  (where  applied),  and  the  judgement  of  the  CPT 
were  virtually  unanimous  in  citing  these  improvement 
recommendations . 

6.1  THE  SIMULATOR  TEST  2000  PROCESS 

The  recommended  new  process  known  as  Simulator  Test  2000 
(ST  2000)  is  shown  in  Figure  6.1-1.  When  contrasted  against  the 
current  test  process  (Appendix  B) ,  the  elimination  of  redundant 
Government  CPSG  testing,  in-plant  performance  tests,  and  on  site 
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acceptance  tests,  plus  a  reduction  in  TRRs,  becomes  obvious.  To 
convey  these  improvements,  the  substance  of  the  ST  2000  process  is 
described  along  with  the  ensuing  recommendations. 

6.1.1  Systems  Performance  Assessment 

A  need  exists  early  in  the  test  program  to  assess  the 
performance  of  the  devices.  The  CPT  recognizes  that  even  with  a 
comprehensive  set  of  data,  it  is  often  incomplete  and  does  not 
always  cover  abnormal  conditions,  malfunctions,  all  weather 
effects,  or  flight  conditions  outside  normal  envelopes.  The  System 
Performance  Assessment  (SPA)  fulfills  this  need  by  providing  SMEs 
to  assist  the  contractor  development  process.  In  implementation, 
the  contractor  will  provide  a  menu  of  areas  to  be  assessed  by  Air 
Force  SME's.  SME  inputs  are  considered  vital  and  are  expected  to 
be  incorporated  by  the  contractor.  Upon  SME  evaluation,  subjective 
inputs  are  quantified  and  incorporated  in  the  DCL,  thus  establish¬ 
ing  an  agreed  to  basis  for  their  incorporation.  Incorporating  SPA 
within  the  ST-2000  process  allows  the  contractor  more  time  in-plant 
to  resolve  training  related  deficiencies.  Improvements  forecasted 
include  a  reduced  test  program  schedule  and  fewer  deficiencies  in 
training  devices  being  fielded. 

6.1.2  Contractor  Engineering  Verification  Tests 

This  segment  of  the  ST-2000  process  represents  a 
significant  improvement  in  the  time  to  accomplish  acceptance 
testing  by  incorporating  the  following  improvements: 

o  Contractor  performance  of  engineering  development 
tests  with  results  completely  documented 

o  Test  procedures  written  at  a  higher  functional  level 

o  Greater  emphasis  on  automated  testing 

o  Automated  error  detection  in  the  software  compile/load 
build  process 

o  Elimination  of  Government  CPSG  tests. 

Contractor  Engineering  Verification  Tests  (CEVT)  are  all 
engineering  tests,  including  automated  Approval  Test  Guides  (ATGs) 
similar  to  those  required  by  FAA  Advisory  Circular  120-40,  required 
by  the  Test  Matrix.  The  tests  are  conducted  by  engineering  and 
witnessed  by  appropriate  Quality  Assurance  personnel.  As  requested 
by  the  contractor,  Air  Force  SMEs  will  be  made  available  to  confirm 
correct  systems  operation. 

6.1.3  Teat  Readiness  Assessment  -  In  Plant 

A  single  in-plant  TRA  supports  the  CPT  recommendation  to 
reduce  the  number  of  TRRs  in  the  current  process.  The  TRA  provides 
the  opportunity  to  review  prior  contractor  test  activity  using 
engineering  and  ATG  test  results  as  documented  evidence  of 
specification  compliance.  This  information,  in  conjunction  with 
the  status  of  contractor  deficiencies,  establish  the  review 
criteria  for  readiness  of  the  device  to  begin  Government  functional 
and  operational  tests. 

6.1.4  System  Performance  And  Demonstration  Evaluation 

The  elimination  of  redundant  testing  was  a  major 
objective  of  the  CPT.  In  the  process  of  gaining  feedback  from 
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military  and  commercial  simulator  programs,  a  strong  emphasis  was 
made  to  assure  that  the  device  was  capable  of  supporting  the 
training  syllabus  prior  to  shipment  from  the  contractor’s  plant. 
The  System  Performance  and  Demonstration  Evaluation  (SPADE) 
satisfies  the  above  intentions  by  performing  functional  check 
flights  or  free  play  using  ATGs  and/or  predefined  mission  scenarios 
for  aircrew  trainers  and  selective  stand-alone  malfunction  tests 
for  maintenance  trainers.  This  testing,  conducted  by  the  Govern¬ 
ment  and  supported  by  the  contractor,  is  the  first  opportunity  for 
the  Government  test  team  to  generate  discrepancies.  It  provides  a 
trainer  evaluation  in  which  discrepancies  that  impact  training  on 
the  device  are  identified  and  resolved.  In  no  case  is  a  repetition 
of  prior  CEVT  advocated. 

6-1.5  Joint  Ready  To  Ship  Assessment 

The  objective  is  to  determine  readiness  to  ship  after 
joint  Contractor/Government  review  of  all  previous  test  results  and 
evaluations.  Criteria  should  include  the  following: 

o  Documented  evidence  of  specification  compliance  is 
acceptable 

o  Previous  contractor  discrepancies  have  been 
dispositioned 

o  SPADE  is  successful  and  discrepancies  affecting 
training  have  been  resolved/corrected,  or  their 
disposition  has  been  agreed  upon 

o  Site  planning  has  been  accomplished. 

6.1.6  on-8ite  contractor  Installation 

This  activity  is  similar  to  that  conducted  in  the  current 
acceptance  process.  The  contractor,  upon  teardown,  pack,  and  ship, 
proceeds  with  on-site  installation  and  checks  the  device  to  insure: 

o  Equipment  operation  is  restored  to  pre-ship  condition 

o  Facility  related  installations  are  functioning 
properly 

o  Any  contractor  deficiencies  which  can  be  corrected  at 
that  time  are  accomplished. 

6.1.7  Facility  and  Safety  interface 

These  tests  are  conducted  by  the  contractor  and  witnessed 
by  the  Government.  The  tests  include  applicable  CEVT,  facility 
installation  and  interface  tests  plus  safety  tests  that  can  only  be 
conducted  as  part  of  final  installation.  Since  these  tests  form 
part  of  formal  device  acceptance,  TDs  may  be  generated.  In 
addition,  the  contractor  can  submit  previous  open  contractor 
discrepancies  that  require  recheck. 

6.1.8  Test  Readiness  Assessment  On-Site 

During  this  TRA,  the  Government  reviews  the  results  of 
contractor  on-site  test  activity  including  installation  and 
checkout,  CEVT,  facility,  safety,  and  discrepancy  status.  The 
final  assessment  that  the  training  device  is  ready  for  Government 
functional  mission  acceptance  tests  is  ultimately  the  Government's 
decision. 
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flovarnmant  Punctional  Mission  Tests 

The  CPT  recognized  that  the  level  and  type  of  testing 
should  be  focused  on  the  device's  viability  for  training.  The  CEVT 
validated  the  soundness  of  the  design  through  engineering  and 
automated  ATG  tests.  A  similar  site  related  contractor  engineering 
test  activity  satisfied  facility  related  requirements.  The  purpose 
of  Government  mission  acceptance  test  is  to  perform  final  accept¬ 
ance  tests  with  emphasis  on  training  capability  and  closure  of 
discrepancies . 

Mission  testing  uses  pre-defined  scenarios  and  free  play. 
Operational  procedures  and  characteristics  are  further  evaluated 
within  the  context  of  conducting  the  mission.  The  Government 
conducts  these  tests  with  contractor  support.  TDs  are  written  with 
closure  authority  and  decision  of  acceptability  of  results  resting 
with  the  Government. 

6 . 2  RECOMMENDATIONS 

The  following  represents  the  list  of  recommendations  made 
by  the  CPT. 


1.  Implement  the  Simulator  Test  2000  process  by  forming  a 
Government  Corrective  Action  Team  to  draft  a  generic  SOW 
and  data/contract  provisions  and  insert  Physical 
Configuration  Audit  requirements  (if  any)  at  the 
appropriate  point  in  the  process.  (Source:  Process 

Analysis) 

a.  Engineering  development  tests  should  be  per¬ 

formed  and  results  submitted  to  the  Government 
for  acceptance  to  eliminate  redundant  testing. 
(Source:  Waste  Areas  2  and  3) 

b.  To  minimize  ambiguous  design  requirements  and 

avoid  the  tendency  to  redesign  during  both 
contractor  and  Government  test  phases,  the 
Government  will  make  SMEs  available  to  assist  in 
subjective  evaluations.  Subjective  data  will  be 
quantified  and  incorporated  in  the  DCL. 
(Source:  Waste  Areas  1,  2,  and  4) 

c.  Greater  emphasis  should  be  placed  in  the  SOW  to 
automate  test  routines  similar  to  FAA  Advisory 
Circular  120-40  requirements  for  commercial 
airlines  to  produce  reliable,  repeatable,  and 
well-documented  results  and  alleviate  the  need 
for  redundant  Government  tests.  (Source:  Waste 
Areas  2  and  5) 

d.  Government  test  procedures  should  be  written  at 
a  functional  level  to  facilitate  SPADE  and  on¬ 
site  mission  testing.  (Source:  Waste  Area  5) 

e.  The  training  system  developer  should  manage  a 
single  discrepancy  status  accounting  system  to 
eliminate  duplicative  and  often  disparate 
systems.  During  test,  an  ad-hoc  team  to 
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identify  and  manage  the  resolution  of  "show 
stopper"  TDs  should  be  established.  (Source: 
Waste  Area  6) 

f.  The  SOW  should  emphasize  automated  error  detec¬ 
tion  in  the  software  compile/load  build  process 

to  drastically  reduce  software  generation 
errors.  (Source:  Waste  Area  6) 

g.  Reduce  t.he  number  of  TRRs  to  minimize  delays  in 
the  test  schedule  by  shifting  the  burden  of  test 
readiness  to  the  contractor.  (Source:  Waste 

Area  7) 

h.  Provide  the  training  system  contractor  the 
opportunity  to  eliminate  Government  CPSG  and 
realize  schedule  and  resource  economics  through 
advanced  Government  certification  of  software 
configuration  management  systems.  (Source: 

Waste  Area  8) 

2.  A  well  defined  and  realistic  Training  Systems 
Requirement  Analysis  should  be  developed  and  released 
prior  to  release  of  the  RFP  to  establish  and  bound 
training  tasks  as  early  as  possible  in  program 
development.  (Source:  Waste  Area  1) 

3.  Make  SMEs  available  to  the  training  device  contractor 
early  in  the  program  to  help  resolve  data  deficiency 
problems  and  establish  consensus  on  interpretation  and 
application.  (Source:  Waste  Areas  1,  2,  and  U) 

f4 .  Implement  the  Simulator  Data  Integrity  Program  study 
recommendations  to  ensure  timely,  accurate,  and  complete 
data  availability  to  the  training  device  developer  from 
the  weapons  system  design  contractor.  (Source:  Waste 
Area  1) 

5.  Establish  a  CPT  to  investigate  solutions  to  incomplete 
Hardware/Software  Integration  planning  including  start, 
stop,  and  in-process  measurement  criteria  to  assure 
improved  management  of  this  important  development  step 
which  is  critical  to  CEVT.  (Source:  Waste  Area  1) 

6.  Mitigate  technical  performance  and  schedule  risks  by 
using  prototype  testing  to  mature  new  technology 
applications  prior  to  attempted  insertion  into  Hardware/ 
Software  Integration.  (Source:  Waste  Areas  1  and  3) 

7-  To  relieve  the  schedule  delays  attributable  to 
unavailable/late  aircraft  components,  early  identifica¬ 
tion  of  requirements  (including  spares)  followed  by 
obtaining  sufficient  priority  for  timely  acquisition 
from  the  weapons  system  contractor  is  considered 
essential.  Components  could  be  manufactured  or  alter¬ 
nately  provided  by  the  training  system  developer  via  an 
associate  contract  agreement  with  the  weapons  system 
developer.  The  use  of  equivalent,  non-military 
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components  (i.e.,  commercial  standards,  designed  for 
foreign  governments)  should  be  allowed  and  encouraged. 
(Source:  Waste  Areas  1  and  6) 

8.  Prime  contractors  must  strengthen  subcontractor/vendor 
management  processes  to  improve  delivery  performance  and 
reduce  the  impact  on  device  readiness.  (Source:  Waste 
Area  1) 

9.  Require  more  aggressive,  comprehensive  performance 
warranties  to  crystal ize  contractor  liability  and 
bolster  Government  confidence  in  contractor  assertions 
to  "meet  the  specification."  This  will  radically  reduce 
contract  test  requirements.  (Source:  Waste  Area  2) 

10.  Contract  specification  test  requirement  definitions  must 
be  clarified  to  assure  current  application  of  test 
techniques  for  performance  validation.  Traditional 
bottoms-up  testing  would  virtually  be  eliminated.  Test 
application  should  be  determined  at  design  reviews. 
Additionally,  MIL- PRIME  Air  Force  Guidance  specifica¬ 
tions  87241  for  flight  simulators  and  87228  for 
maintenance  training  devices  requires  revisions  to 
include  test  definitions.  (Source:  Waste  Areas  3  and 
5) 

11.  Training  of  contractor  test  personnel  in  the  principles 
of  TQM  would  greatly  improve  product  quality.  Reduction 
in  discrepancies  and  a  more  thorough  accomplishment  of 
procedures  would  practically  be  guaranteed.  (Source: 
Waste  Areas  4,  5,  and  6) 

12.  CDRL  responsibilities  should  be  centralized  with  the 
Government  lead  engineer  to  avoid  duplication  and 
unnecessary/uncoordinated  test  documentation 
requirements.  (Source:  Waste  Area  5) 

13.  Institute  a  program  to  allow  pre-approval  of  contractor 
plans  required  for  Government  oversight  to  eliminate  the 
need  for  separate  plans  for  each  program.  (Source: 
Waste  Area  5) 

14.  To  abate  the  impact  of  test  interruptions,  risk  manage¬ 
ment  programs  must  be  developed  and  implemented  to 
anticipate  and  manage  possible  causes.  (Source:  Waste 
Area  6) 

15.  A  repair  pipeline  should  be  negotiated  with  the  original 
equipment  manufacturer  prior  to  testing  to  insure  the 
availability  of  spares  during  the  test  program. 
(Source:  Waste  Area  6) 

16.  Training  device  development  programs  requiring  new / 
modified  facilities  should  consider  tasking  the  develop¬ 
ment  contractor  as  authorized  by  10  USC  2353  to 
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centralize  contract  engineering  responsibilities. 
(Source:  Waste  Area  6) 

17.  Remove  test  duplication  that  exists  in  software  at  the 
Computer  Program  Component  and  Computer  Program  Configu¬ 
ration  Item  level  (between  2167A  and  acceptance  test 
requirements)  by  revising  2167A  criteria.  (Source: 
Waste  Area  2) 

18.  As  a  factor  in  the  procurement  selection  process, 
develop  a  Blue  Ribbon  Contractor  List  based  upon  the 
contractors  test  track  record.  (Source:  CPT  expertise. 
See  paragraph  6.3.1) 

19.  Test  incentive  awards  should  be  applied  to  contracts  and 

structured  as  a  strong  motivator  as  in  other  critical 
performance  areas.  (Source:  CPT  expertise.  See 

paragraph  6.3.2) 

20.  Minimize  the  value  associated  with  the  contractual 
milestone  for  shipment  to  reduce  conflict  between  test 
completion  and  ship  milestones.  (Source:  CPT  exper¬ 
tise.  See  paragraph  6.3.3) 

6.3  CPT  GENERATED  RECOMMENDATIONS 

6.3.1  Test  Blue  Ribbon  Contractor  program 

The  CPT  endorses  development  of  an  additional  contract 
provision  to  reward  superior  contractor  test  performance.  The 
purpose  is  to  formalize  waiving  test  program  past  performance 
assessment  requirements  once  a  contractor  demonstrated  exceptional 
compliance  to  contract  requirements.  This  provision  recognizes 
that  among  responsible  contractors,  varying  degrees  of  quality  and 
test  programs  exist. 

An  integral  part  of  this  program  is  the  development  of  a 
Blue  Ribbon  Contractor  list  for  simulation  and  training  device 
manufacturers  participating  in  either  firm  fixed  pricing  or  cost 
plus  FSD  contracts.  Membership  on  this  list  indicates  a  contractor 
has  demonstrated  dependable  quality  and  test  program  performance  on 
AFSC  contracts  during  the  past  year.  The  Blue  Ribbon  Contractor 
List  will  be  a  major  factor  in  the  evaluation  of  past  quality  and 
test  program  performance. 

The  suggested  criteria  for  membership  on  a  Blue  Ribbon 
Contractor  List  are  as  follows: 

1.  The  contractor  has  been  awarded  and/or  delivered  items 
on  at  least  one  Air  Force  Systems  Command  (AFSC)  FSD 
contract  worth  an  aggregate  total  of  $10M  or  more 
during  the  past  three  years,  and, 

2.  The  contractor  has  demonstrated  90  percent  or  higher 
on-time  delivery  (total  quantity  delivered  on-time 
divided  by  total  quantity  due)  of  all  test  related 
data  items  of  exceptional  quality  (first  time 
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submittal  gained  Government  approval  with  only  minor 
correction  required) . 


3.  The  contractor  demonstrated  quality  performance  on 
AFSC  contracts  measured  as  follows: 

-  Resolution  and  rapid  correction  of  priority  TDs 
must  have  been  accomplished  within  45  calendar 
days  or  less.  This  includes  completion  of  all 
Government  in-plant  testing  through  total  cor¬ 
rection  (to  the  Government's  satisfaction)  of 
all  Priority  1  and  2  TDs.  For  purposes  of 
discussion,  the  45  day  duration  is  measured  from 
the  day  of  discovery  and  submittal  ("Day  1") 
through  successful  recheck  of  all  Priority  1  and 
2  TDs  ("Day  45  or  less") .  The  various  TD  prior¬ 
ities  are  defined  as  follows: 

Priority  1:  Those  TDs  involving  safety,  which 
require  immediate  correction  in  order  to  prevent 
injury  to  test  personnel  or  to  prevent  damage  to 
equipment . 

Priority  2 :  Those  TDs,  other  than  safety,  which 
if  not  corrected  could  result  in  an  extension  of 
the  planned  test  schedule  or  which  may  impact 
training. 

4.  No  known  in-plant  Quality  Deficiency  Reports  (QDRs) 
(i.e.,  unresolved  method  C,  D,  or  E)  are  permitted. 
A  QDR  is  defined  as  a  report  generated  due  to  a  defect 
which  is  found  during  acceptance  or  visual  inspection 
or  during  functional  test  or  checkout  which  indicates 
a  failure  or  that  failure  is  imminent,  and  is  attrib¬ 
utable  to  errors  in  workmanship  or  nonconforming 
material.  QDRs  are  generated  from  field  activities 
(i.e.,  DCAS ) . 

Membership  on  the  Test  Blue  Ribbon  Contractor  List  is  by 
application  only.  Any  contractor  that  meets  the  criteria  described 
above  may  apply  for  membership  by  providing  the  following  informa¬ 
tion  to  ASD/YWK. 

The  number  of  AFSC  contracts  received  in  the  past  three 
years  to  include  the  following  information: 

a.  Contract  number. 

b.  Date  of  award. 

c.  Total  contract  price. 

d.  Type  of  testing  accomplished  and  duration. 

e.  Significant  test  accomplishments. 

At  least  semi-annually  The  Training  Systems  SPO  will 
convene  a  panel  to  review  contractor  applications  for  membership  on 
the  Test  Blue  Ribbon  Contractor  List.  The  panel  will  use 
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Government  data  to  validate  a  contractor's  application.  Data  will 
include,  but  not  be  limited  to:  TD  Management  System,  Data  Item 
Tracking  System,  and  Contract  Administration  Office  maintained 
data,  including  the  Quality  Assurance  Representative's  documented 
records  of  a  contractor's  in-plant  Quality  Program.  Those 
contractors  meeting  or  exceeding  the  membership  criteria  will  be 
placed  on  the  Test  Blue  Ribbon  Contractor  List. 

Contractors  on  a  Blue  Ribbon  Contractor  List  will  be 
reviewed  semi-annually  to  ascertain  whether  or  not  their  qual  ity 
and  test  program  performance  still  meet  the  criteria  for  Blue 
Ribbon  membership.  One  year's  worth  of  data  will  be  reviewed. 
Contractors  who  no  longer  meet  the  criteria  will  be  deleted  from 
the  Test  Blue  Ribbon  Contractors  List.  The  contractor  will  be 
notified  in  writing  of  the  deletion  and  the  reasoning  of  the  panel. 

Contractors  whose  names  appear  on  the  Blue  Ribbon 
Contractor  List  do  not  need  to  submit  past  performance  data  with 
any  proposals  responding  to  RFPs.  In  addition.  Contractor 
Performance  Assessment  Reports  as  required  by  AFSC  regulation 
800-54  will  require  no  evaluation  in  the  test  and  evaluation  area 
for  those  qualified  contractors.  Simply  inserting  "Test  Blue 

Ribbon  Contractor"  will  suffice.  The  CPT  urges  this  be  implemented 
at  the  earliest  possible  opportunity  with  concurrence  of  the 
Training  Systems  SPO  Program  Director. 

6.4.2  Contractor  Teat  Performance  Incentives 

The  CPT  recognized  that  the  contractor  currently  has 
every  incentive  to  start  Government  test  to  see  if  he  can  "sell- 
off"  the  device  and  save  schedule.  If  testing  fails  to  achieve  the 
desired  result,  the  contractor  may  find  it  more  economical  to 
resist  corrections,  attempt  short  term  solutions,  and  hope  test 
schedule  concerns  will  cause  the  Government  to  weaken  its  position. 

The  CPT  believes  that  c*:-cr>bional  contractor  test 

performance  should  ue  rewarded.  Conduct  of  an  effective,  well 
planned  test  program  is  a  worthwhile  objective.  The  creation  of 
contract  incentives  to  accomplish  this,  however,  is  dependent  upon 
several  variables  including  the  basic  nature  of  the  testing 
(development  vs.  production) ,  the  type  of  contract  (cost  vs.  fixed 
price),  and  the  type  of  incentive  (i.e.,  objective  performance 
incentive  versus  subjective  award  fee  incentive) .  Each  of  these 
factors,  along  with  other  pertinent  facts,  must  be  weighed  when 
trying  to  assess  the  ability  to  create  a  "real"  contract  motivator. 
The  ultimate  question  to  be  satisfied  is  "What  motivates  the 
contractor?"  Is  it  profit,  sales,  cash  flow,  etc.?  The  answer 
lies  in  the  contract  type  which  is  what  inspires  motivation. 

o  Development  Testing  -  Cost  Type  Contract  -  Award  Fee 
Because  of  the  very  nature  of  a  development  program, 
it  would  be  extremely  difficult  to  structure  a 
performance  incentive  which  would  be  meaningful. 
Development  testing  by  its  very  nature  is  intended  to 
surface  problems  before  the  design  is  frozen  and  moved 
into  a  production  and  deployment  environment.  The  key 
is  solid  test  planning  and  analysis  in  order  to 
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minimize  surprises.  These  tend  to  be  much  more 
subjective  areas  when  attempting  to  establish  measure¬ 
ment  criteria.  Additionally,  if  work  is  being 
performed  under  a  cost  type  contract,  the  need  to  get 
the  hardware  bought  off  in  order  to  liquidate  cost  and 
gain  profit  is  nonexistent.  Award  fee  provisions 
would  allow  tailoring  of  the  incentive  from  period  to 
period  as  the  program  progresses  and  provides  multiple 
opportunities  to  comment/ reward  performance 

o  Production  Acceptance  Testing  -  Fixed  Price  Type 
Contract 

-  Performance  Incentive 

In  a  production/deployment  environment,  test  require¬ 
ments  have  been  defined.  Because  of  this,  the 
Government's  ability  to  write  a  meaningful  incentive 
at  the  time  of  contract  award  is  much  greater.  A 
concern  remains  that  the  incentive  being  structured  is 
sufficient  in  terms  of  dollars  or  corporate  visibility 
to  be  an  effective  motivator  and  is  in  balance  with 
the  remainder  of  the  program.  From  a  cash  flow/ 
liquidation  standpoint,  the  contractor  is  still 
motivated  to  push  for  contract  buy  off  in  order  to 
claim  progress  payments  and  profit.  Additionally,  the 
incentive  must  be  justifiable  in  terms  of  overall 
savings  to  the  Government  (i.e.,  reduced  Temporary 
Duty,  more  efficient  use  of  personnel,  reduced 
paperwork,  etc.). 

The  CPT  recommends  that  initiatives  begun  by  The  Training 
Systems  SPO  should  be  continued  and  applied  to  all  new  acquisition 
programs.  It  is  extremely  important  to  monitor  results  of  programs 
where  incentives  have  been  applied  to  ensure  the  level  of  value  is 
worthwhile  to  the  contractor,  that  they  remain  realistic  and 
achievable,  and  experience  gained  through  their  application  is 
reinvested  in  future  programs. 

6.4.3  Milestone  Funding 

Existing  Firm  Fixed  Price  contract  funding/billing 
milestone  schedules  may  influence  acceptance  test  performance.  Too 
much  financial  value  is  associated  with  the  shipment  of  training 
devices  from  the  contractor's  facility.  This  places  significant 
pressure  on  the  test  organization  to  start  Government  acceptance  as 
soon  as  possible  and  ship  before  all  problems  have  been  corrected 
and  re-tested  in  order  to  capitalize  on  contract  milestone  payment 
provisions.  It  causes  the  contractor  to  be  willing  to  "take  the 
risk"  that  problems  can  be  solved  during  second  shift,  in  the 
period  of  equipment  tear  down,  pack,  and  ship,  or  during  site 
installation  and  checkout. 

Having  the  contractor  revenue  events  place  less  emphasis 
on  the  ship  milestone  could  improve  overall  test  performance.  The 
contractor  would  keep  the  equipment  in  plant  to  complete  problem 
resolution  and  re-test.  This  approach  enables  the  contractor  to 
conduct  testing  in-plant  which  is  much  more  cost  effective  to  both 
the  contractor  and  Government.  The  contractor  can  solve  problems 
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quicker  by  having  all  his  resources  available  rather  than  using 
'•tiger  teams"  on-site. 

The  CPT  recommendation  is  to  reduce  the  size  of  the 
dollar  value  associated  with  training  device  shipment  from  the 
contractors  facility.  The  decision  to  start  acceptance  testing 
should  not  be  influenced  by  a  large  contractor  revenue  event.  The 
final  contract  funding  profile  should  place  more  emphasis  on  final 
acceptance  testing  on-site.  The  contractor  would  still  be 
motivated  to  finish  as  soon  as  possible  but  is  not  incentivizea  to 
prematurely  ship  the  device. 

6.4  PROCESS  IMPROVEMENT  BENEFITS 

For  any  recommended  process  change  to  be  considered  for 
implementation,  a  measurable  improvement  must  be  expected.  If 
there  is  a  significant  anticipated  benefit  as  a  result  of  the  new- 
process,  a  high  degree  of  motivation  to  adopt  the  new  process  will 
be  present.  The  consensus  of  this  CPT  is  that  with  the  adoption 
and  implementation  of  ST-2000  process,  a  significant  savings  in 
resources  can  be  achieved.  Additionally,  if  the  recommendations 
and  other  categories  of  suggested  changes  which  are  not  directly 
part  of  the  formal  simulator  test  phase  are  also  implemented, 
additional  efficiency  in  each  training  device  acquisition  program 
can  be  achieved. 

A  comparison  of  the  total  contractoi  and  Government  test 
effort,  referred  to  as  the  "Idealized  WST  Test  Program",  can  be 
made  to  the  ST-2000  process  test  program.  The  Idealized  WST  Test 
Program  assumes  that  once  testing  begins,  it  progresses  and  is 
completed  without  delays  or  interruptions.  This  test  program 
includes  CVT,  Performance/Acceptance  Tests,  and  in-plant  as  well  as 
on-site  Operational  Evaluations  as  depicted  below: 


Idealized  WST  Test  Program 
IN-PLANT  ON-SITE 

Performance  OPS  Acceptance  OPS 


TESTS : 

CVT 

Test 

EVAL 

Test 

EVAL 

TOTAL 

WEEKS : 

20 

18 

2 

6 

2 

48 

The  total  test  effort  of  forty-eight  (48)  weeks  required 
by  the  idealized  WST  test  program  can  now  be  compared  to  estimates 
for  WST  testing  using  the  ST-2000  process. 
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ST  2000  PROCESS 


IN-PLANT 

ON-SITE 

SYSTEM 

FUNCTIONAL 

TESTS : 

ASSESSMENT  CEVT 

SPADE 

&  MISSION  TESTS 

TOTAL 

WEEKS: 

1.5  20 

6 

3 

30.5 

The  reduction  of  the  total  test  effort  from  forty-eight 
(48)  weeks  to  thirty  and  one  half  (30.5)  weeks  represents  a  savings 
of  approximately  thirty-seven  (37)  percent.  The  CPT  members' 
consensus  is  that  this  amount  of  savings  can  be  realistically 
expected  if  all  elements  of  ST-2000  process  are  implemented. 

The  CPT  also  made  several  recommendations  and  other 
observations  which  pertain  to  the  total  training  device  acquisition 
program.  These  are  not  within  the  formal  test  phase,  but  directly 
affect  the  start  or  progress  of  testing.  The  potential  savings  to 
cost  and  schedule  which  can  be  achieved  by  implementing  these 
changes  are  not  overlooked.  The  largest  waste  in  most  military 
training  device  development  programs  occurs,  for  a  variety  of 
reasons,  prior  to  the  device  being  ready  for  testing.  A  review  of 
the  Test  Program  Overview,  Days  Late  At  Milestones  (Military) ,  data 
shows  that  on  average  military  programs  are  158  days  (32  work 
weeks)  late  prior  to  beginning  of  test.  The  elimination  of  this 
waste  would  result  in  cost  and  schedule  overrun  savings  of 
approximately  twenty  (20)  percent  over  the  life  of  a  planned 
thirty-six  (36)  month  program. 
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7.0 


CONCLUSION 


The  YW/Industry  CPT  spent  one  year  engaged  in  the 
analysis  of  the  current  training  device  acceptance  test  process  and 
the  development  of  improvements.  Using  the  Cumberland  Group's 
Process  Improvement  Model  to  structure  the  team's  activities, 
monthly  working  sessions  were  used  to  manage  the  effort.  A 
conservative  estimate  of  time  expended  by  each  team  member  is  in 
excess  of  400  man  hours. 

With  the  completion  of  the  team's  assignment,  the 
resulting  improvement  recommendations  clearly  indicate  that  success 
was  achieved.  Time  was  indeed  well  spent,  resources  were  adequate, 
and  the  potential  payback  without  any  compromise  to  training  device 
performance  is  substantial . 

To  measure  success,  the  CPT  reviewed  accomplishments  with 
respect  to  the  improvement  goals  established  at  the  commencement  of 
the  effort  as  discussed  in  Section  2.0,  Definition  of  the  Improve¬ 
ment  Opportunity.  In  formulating  these  goals,  the  CPT  focused  on 
several  fundamental  principles  considered  essential  to  whatever 
process  changes  might  be  recommended  to  improve  acceptance  testing 
of  aircrew  and  maintenance  trainers.  These  principles  generally 
promote  the  concept  that  quality  must  be  enhanced  while  time 
required  for  testing  is  reduced.  The  following  paragraphs 
highlight  the  relationship  between  study  goals  and  the  most 
prominent  recommendations. 

GOAL  #1.  MINIMIZE  THE  NUMBER  OF  TEST  ACTIVITIES  WITHOUT 
COMPROMISING  PERFORMANCE  VALIDATION.  While  reviewing  the  present 
acceptance  test  process,  the  CPT  identified  a  number  of  contractu¬ 
ally  required  test  activities  that  seemed  to  add  little  value,  were 
redundant,  or  that  could  be  accomplished  jointly  by  the  contractors 
and  Government.  Several  areas  of  contractor  testing  repeated  by 
the  Government,  multiple  TRRs,  and  numerous  delay  paths  were 
identified  through  process  analysis.  Held  sacrosanct  was  that 
reduction  in  test  activity  could  only  be  recommended  provided  that 
performance  validation  could  not  be  bargained  for  any  reductions  in 
test  activity.  The  customer  was  to  be  assured  that  the  training 
device  has  been  adequately  tested  to  guarantee  the  quality  and 
performance  required  by  specification.  Predominate  recommendations 
for  improvements  include  the  ST  2000  Test  Process,  waiving  CPSG, 
prototype  testing  of  new  technology,  and  the  expanded  use  of 
contract  warranty  and  incentive  provisions. 

GOAL  #2.  FOSTER  TIMELY.  ACCURATE.  STREAMLINED  DEVELOP¬ 
MENT  OF  TEST  DOCUMENTS.  The  data  collected  showed  a  wide  diversity 
of  test  procedures  written  at  various  levels  of  comprehension  and 
detail  that  do  not  correlate  with  the  complexity  of  individual 
training  devices.  The  team's  concern  that  the  development  of  test 
documents  may  be  delayed  due  to  the  Government  review  and  approval 
process  and  that  the  documents  are  written  to  a  level  of  detail 
necessary  only  for  inexperienced  test  personnel  was  reinforced.  If 
test  document  development  could  be  streamlined  and  standardized, 
the  possibility  of  reducing  the  number  of  formal  test  days  could  be 
a  realistic  achievement.  Centralized  data  responsibilities. 
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improved  test  requirement  (matrix),  automated  test  routines,  and 
test  procedures  written  at  the  functional  level  are  major  enhance¬ 
ments  toward  satisfying  this  goal.  Certification  of  certain 
contractor  test  processes  by  implementing  a  system  for  pre-approved 
plans  also  goes  a  long  way  to  sort  out  the  test  document  quagmire. 

GOAL  #3.  FACILITATE  GOVERNMENT  ACCEPTANCE  OF  QUALIFIED 
SQgEB&STQR.  TI8.E  .RSgq.LT9 .  Few  would  question  that  schedules  are 
extended  when  tests  are  repeated  by  the  Government  even  when 
qualified  contractor  test  results  may  be  available.  Major  trainer 
systems  are  often  retested  and  subsystem  engineering  tests  repeated 
due  to  mistrust  and  a  lack  of  documented  test  results.  This 
typifies  the  "business-as-usual”  syndrome  in  the  well  established 
process  and  illustrates  the  traditional  mindset  of  bottoms-ur 
testing.  To  break  this  cycle,  allocating  clear  test  and  reporting 
responsibilities  to  the  contractor  is  essential.  Agreements  on 
format,  content,  and  other  issues  relating  to  acceptability  cf 
results  must  be  negotiated  between  training  system  contractors  and 
the  Government.  Major  improvement  recommendations  to  require  the 
contractor  to  conduct  engineering  development  tests,  write  and 
resolve  discrepancies,  and  revamp  the  Hardware/Software  Integration 
phase  of  program  development  are  all  essential  to  strengthen  the 
Government's  belief  in  contractor  results. 

GOAL  #4.  TAILOR  APPLICATION  OF  MIL-SPEC/STD  TEST 
REQUIREMENTS  TO  BMPHA8IZE  COMMERCIAL  PRACTICES.  The  fundamental 
concept  is  to  test  only  what  needs  to  be  tested  at  the  level  which 
will  insure  specification  compliance.  This  is  often  difficult  to 
describe  because  there  is  no  single,  universally  accepted  defini¬ 
tion  of  what  constitutes  "Best  Commercial  Practices”.  For  t."e 
purpose  of  this  study,  the  CPT  agreed  that  commercial  practices 
were  simply  methods  of  doing  business  used  by  commercial  operators 
that  made  sense.  Primary  commercial  practices  recommended  include 
the  implementation  of  performance  warranties  to  relax  the  need  to 
accomplish  detailed  subsystem  testing  and  the  development  ot 
standards  for  obtaining  accurate  performance  data  packages  for  use 
by  the  system  developer.  Commercial  facility  construction 
practices  should  be  pursued  to  simplify  matching  facility  design 
and  schedule  considerations  with  those  of  the  training  device 
development.  Yet  another  recommendation  towards  satisfying  this 
all  important  objective  is  to  use  ATGs  similar  to  those  required  by 
the  FAA  in  their  certification  of  flight  simulators  for  aircrew 
training.  Tailoring  MIL-STD-2167  requirements  to  eliminate 
repetitive  testing  is  also  strongly  advocated  and  can  be  accom¬ 
plished  now  by  the  buying  agency. 

GOAL  #5.  ENCOURAGE  IMPROVEMENT  OF  ON-TIME  TEST  MILESTONE 
PERFORMANCE .  Strong  evidence  of  waste  is  clearly  obvious  from  the 
available  data  serving  to  underscore  the  appropriateness  of  this 
goal.  The  capability  of  the  contractor  to  meet  schedule  commit¬ 
ments  during  the  development  (pre-test)  phase  is  vital  in  the 
overall  improvement  of  on-time  test  milestone  performance.  The 
data  suggests  that  testing,  even  when  begun  late,  tends  to  follow 
the  planned  schedule.  Recommendations  to  eliminate  roadblocks  to 
on-time  testing  are  numerous.  Timely  availability  of  products, 
documenting  training  requirements,  integration  of  Government  SMEs 
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in  the  development  process,  resolution  of  subjective  data  areas, 
identification  and  ordering  of  aircraft  components,  and  a  genuine 
test  management  risk  abatement  plan  comprise  a  sample  of  those 
available.  Possibly  more  significant  are  those  incentive/ award 
improvements  which  include  financial  considerations  for  exceptional 
contractor  test  performance. 

GOAL  #6.  REFINE  THE  TEST  DISCREPANCY  GENERATION. 
TRACKING.  AND  RESOLUTION  PROCESS.  TDs  are  generated  during 
acceptance  testing  because  of  a  variety  of  reasons.  Notwithstand¬ 
ing,  TDs  constitute  waste  by  definition  (failure  to  do  it  right  the 
first  time) .  The  late  receipt  of  data  and  the  correct  interpreta¬ 
tion  of  data  both  seem  to  be  factors  which  create  conditions 
causing  the  generation  of  many  TDs.  Inadequate  contractor  testing 
and  test  procedures,  often  due  to  a  lack  of  system  knowledge, 
create  problems  late  in  the  development  cycle  and  their  resolution 
usually  affects  the  test  schedule  Recommendations  to  moderate  the 
impact  of  deficiencies  include  the  use  of  SMEs  early  in  the 
development  program  to  interpret  and  clarify  data,  ad-hoc  reams  to 
prioritize  TDs  for  resolution,  ai d  automated  software  error 
detection  tools  in  the  compile/load  build  process.  A  related 
recommendation  for  the  training  of  contractor  test  personnel  in  the 
principles  of  TQM  to  improve  product  quality  would  as  a  natural 
consequence  reduce  discrepancies. 

COAL  #7.  PROMOTE  GOVERNMENT/INDUSTRY  COOPERATION  AND 
PARTNERSHIP  FOR  IMPROVING  AIRCREW/MAINTENANCE  TRAINING  DEVICE 
TESTING*  One  of  the  best  ways  to  solve  problems  and  thereby 
improve  processes  is  to  motivate  all  parties  to  be  involved  and 
work  together.  The  YW/Industry  Total  Quality  Executive  Steering 
Committee,  formed  in  July,  1989,  by  Col.  Wayne  Lobbestael,  Training 
Systems  SPO  Program  Director  and  his  staff,  certainly  is  the 
genesis  for  the  type  of  Government/ industry  partnership  that  will 
be  effective  in  solving  mutual  problems  to  the  benefit  of  both 
partners.  The  Critical  Process  Team  was  chartered  by  the  Steering 
Committee  to  search  for  opportunities  to  improve  aircrew/mainte¬ 
nance  training  device  testing.  All  recommendations  give  everyone 
a  stake  in  the  outcome  and  each  demands  a  climate  of  pride, 
professionalism,  excellence,  and  trust  to  make  them  work. 

The  premier  improvement  to  simulator  and  maintenance 
trainer  test  process  has  been  identified  as  Simulator  Test  2000. 
Reforms  to  reduce  Government  test,  strengthen  and  reallocate 
contractor  test  responsibilities,  refine  test  documentation,  and 
discrepancy  management  encompass  eight  major  CPT  recommendations. 

The  most  significant  process  change  is  the  customer's 
agreement  to  accept  the  test  results  from  CEVT.  Repetition  of 
these  tests  by  the  Government  is  no  longer  a  requirement.  This 
commitment  eliminates  the  single  largest  cycle  of  testing  from  the 
current  acceptance  test  process. 

In  point  of  fact,  ST  2000  places  the  responsibility  to 
thoroughly  execute  CEVT  squarely  on  the  contractor.  it  must  be 
conducted  to  the  same  level  as  required  for  developmental  perform¬ 
ance  testing  previously  conducted  by  the  Government.  This  means 
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that  test  results  must  be  documented,  verifiable  and  repeatable. 
Failure  to  execute  stringent,  valid  testing  with  documented  results 
will  motivate  the  customer  to  demand  a  repeat  of  previously  run 
tests  and  will  again  require  100%  witnessing  of  CEVT.  if  the 
contractors  do  not  perform  their  part,  the  customer  has  no 
alternative  but  to  revert  back  to  the  existing  test  philosophy. 
The  customer  has  extended  the  opportunity,  the  contractor  must 
aggressively  respond  for  ST-2000  to  be  viable. 
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WILLIAM  J.  ADAMS  BIOGRAPHICAL  DATA 


William  J.  Adams  is  presently  the  Associated  Director  of  Systems 
Engineering  for  Hughes  Simulation  Systems,  Inc.  Mr.  Adams  is 
responsible  for  Visuals,  Operator  Trainers,  Maintenance  Trainers 
and  Naval  Combat  Systems.  Mr.  Adams  has  direct  experience  in 
development  and  sell-off  testing  in  each  of  these  product  areas. 

Design  and  development  of  the  Torpedo  Handling  System  for  the  DD 
963  Spruance  Class  Ship  was  the  first  exposure  to  the  Acceptance 
test  process.  This  activity  covered  development  testing  through 
sell-off  aboard  ship.  Test  sell-off  included  both  a  Prime  and  the 
U.S.  Navy.  This  was  followed  by  the  assignment  to  develop  the 
Modular  Combat  System  which  is  a  fully  distributed  computing  Fire 
Control  System.  As  part  of  this  responsibility  Mr.  Adams  not  only 
directed  total  System  acceptance  but  also  lead  the  Acceptance  Teams 
for  major  subcontractor  eguipment;  HR-76  Track  Radar,  SPS-53 
Surveillance  Radar;  UPA-62  Display  Systems,  totaling  in  excess  of 
$50  million.  Recent  assignments  include  the  management  of  the  F-15 
and  F-16  maintenance  Trainers  and  hands  on  Acceptance  test  of  the 
GBU-15  Part  Task  Trainer,  (Visuals  and  Database) . 

Mr.  Adams  graduated  from  Loyola  University  of  Los  Angeles  in 
January  1967  with  a  Bachelor  of  Science  degree  in  Mechanical 
Engineering.  He  has  an  MBA  in  Finance  from  University  of  Southern 
California,  1971  and  Masters  in  Computer  Science  from  West  Coast 
University,  1978. 
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A.F.  EMERSON  BIOGRAPHICAL  DATA 


Mr.  Emerson  is  presently  the  Vice  President  of  Quality  for  ECC 
International  Corp.  where  he  has  been  employed  since  1973.  since 
that  time  he  has  established  and  managed  the  Integrated  Logistic 
Support,  Reliability  and  Maintainability  Engineering,  Customer 
Service,  Technical  Publications  and  Training  Departments. 

Prior  to  joining  ECC,  Mr.  Emerson  had  extensive  experience  in 
Electrical  Engineering  -  University  of  Maryland;  Reliability 
Engineering  -  General  Electric  Co.;  Quality  Control  Engineering  - 
Spartan  Electronics;  Quality  Management  -  Marine  Resources; 
Reliability  and  Quality  Management  -  Fourdee  Inc. 
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GERALD  KOSYDAR  BIOGRAPHICAL  DATA 


Gerald  M.  Kosydar  has  over  26  years  of  experience  in  all  aspects  of 
simulation  and  training  systems.  He  currently  is  Director  - 
Advanced  Engineering  and  Technology  for  Systems  and  Products  at 
CAE-Link  corporation's  Link  Flight  Simulation  Division. 

Prior  to  this  assignment,  Mr.  Kosydar  was  Director  Engineering  - 
Special  Programs,  which  included  directing  the  engineering  effort 
for  the  Link  B-2  Training  Systems  Program.  Other  prior  assignments 
include  Director  -  Air  Force  Advanced  Simulation  Technology  and 
Director  -  Commercial  Engineering  Technology  where  he  directed 
engineering  development,  including  the  Micro-Simulation  Technology 
program  that  introduced  microprocessor  and  VLSI  technology  into  the 
Link  simulation  product  line. 

Mr.  Kosydar  also  directed,  managed,  and  was  responsible  for  the 
Advanced  Simulation  Technology  program  which  introduced  Large  scale 
Integration  and  Advanced  Integrated  Technology  into  both  Link 1 s 
military  and  commercial  product  lines. 

Other  Engineering  assignments  included:  Manager  -  Systems  and 
Integration,  Manager  -  Computer  and  Instructional  Systems,  Manager 
-  Advanced  Tactics  Development,  and  Manager  -  Special  Products, 
which  included  motion  systems,  visual  systems,  and  computer 
interface  and  I/O  development. 

Earlier  Mr.  Kosydar  was  the  Lead  Engineer  for  the  tactics  portion 
of  F-4  simulators,  and  designed  radar  and  tactical  simulations  for 
the  F-lll,  J-35 ,  and  development  designs  for  new  programs. 

Before  joining  Link,  Mr.  Kosydar  was  responsible  for  advanced 
development  engineering  in  the  ASN/USW  field  and  design  engineering 
of  the  test  facility  for  the  F-111A  inertial  reference  system  for 
General  Dynamics. 

Mr.  Kosydar  spent  uhree  years  in  the  Navy  specializing  in  Radar 
Electronics.  He  holds  a  B.S.  degree  in  Physics  from  the  University 
of  Scranton. 
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ROBERT  A.  LILLIE  BIOGRAPHICAL  DATA 


Mr.  Lillie  is  presently  the  Chief  Engineer  for  the  Tactical/ 
Trainers  Programs,  Training  Systems  SPO.  His  responsibilities 
include  supervising  program  development,  training  requirements 
definition,  design,  acquisition,  and  testing  of  simulators  and 
training  systems  for  Tactical  and  Air  Training  Commands. 

EXPERIENCE: 

Nov  1979  -  Dec  1989:  Lead  Engineer  for  the  F-16  Weapons  System 

Trainer  (WST) .  Responsible  for  the 
technical  efforts  associated  with  the 
design  and  test  of  the  flight  simulator 
and  visual,  EW,  and  DRLMS  subsystems. 
Planned  and  conducted  testing  on  all  F-16 
WST  configurations  and  integration 
testing.  Tested  all  Foreign  Military 
Sales  versions  of  the  F-16  WST. 

Jun  1973  -  Nov  1979:  Aerospace  Engineer.  Provided  functional 

engineering  support  to  various  simulator 
programs.  Analyzed  aerodynamic  data 
requirements,  worked  with  flight  test 
organizations,  monitored  aerodynamic 
modeling,  and  performed  qualification  and 
acceptance  testing  of  T-37,  T-38,  c-130, 
B-52 ,  and  KC-135  simulators. 


SAM  MALENE  BIOGRAPHICAL  DATA 


Mr.  Malene  is  employed  at  Loral  Defense  Systems  Division  in  Akron, 
Ohio.  He  has  been  with  the  company  for  15  years  working  in  various 
simulator  engineering  capacities.  Prior  to  that,  he  served  in  the 
U.S.  Navy  flying  patrol  aircraft.  His  present  position  is  Deputy 
Program  Director  for  the  F-15E  Weapon  Systems  Trainer.  In  this 
capacity,  he  has  overseen  the  test  and  checkout  of  three  highly 
complex  and  sophisticated  training  devices.  Prior  to  this 
assignment,  he  was  the  project  engineer  for  two  F-15J  Flight 
Simulators  for  the  Japanese.  He  has  worked  as  a  development  and 
test  engineer  and  was  involved  with  system  checkout  and  test  on  six 
F-15  Operational  Flight  Trainers  for  the  U.S.  Air  Force.  He  also 
has  helped  install  several  simulators  and  numerous  modifications  to 
field  sites.  He  has  additional  experience  working  with  automatic 
test  equipment  to  check  electronic  circuit  cards  used  in 
simulation. 

Mr.  Malene  has  a  B.S.  in  Electrical  Engineering  from  the  University 
of  Dayton,  and  M.B.A.  from  the  University  of  Akron  and  is  a 
graduate  of  the  Program  Management  Course  at  Defense  Systems 
Management  College  at  Ft.  Belvoir,  VA. 
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LAWRENCE  L.  STEWART  BIOGRAPHICAL  DATA 


Lawrence  L.  Stewart  is  Manager  of  Systems  Test  at  the  Link  Flight 
Simulation  Division  (LFSD)  of  CAE-Link,  Binghampton,  New  York.  Mr. 
Stewart  has  been  continuously  employed  within  the  LFSD  integration 
and  test  organization  where  he  currently  manages  test  and  delivery 
of  helicopter  and  fixed  wing  systems  including  F-4,  C-130,  UH-60, 
AH-64,  and  software  support  centers. 

Mr.  Stewart  is  a  1965  graduate  of  the  U.S.  Naval  Academy,  with  a  MS 
in  Industrial  Management  from  Rensselaer  Polytechnic  Institute. 
His  Navy  training  included  Naval  Flight  Training  and  graduation 
from  the  Naval  War  College,  Newport,  Rhode  Island.  Mr.  Stewart's 
twenty  year  Naval  Aviation  career  included  fleet  and  training  tours 
in  P3  Patrol  Aircraft,  training  aircraft,  and  the  USS  LEXINGTON. 
Overseas  duty  extended  from  Vietnam  to  the  Mediterranean,  North  and 
South  Atlantic,  and  Central  America.  He  has  extensive  training 
experience  in  Navy  primary  aircraft  and  simulation,  including 
assignment  as  a  Quality  Assurance  Representative  and  Contracting 
Officer’s  Technical  Representative  during  his  last  three  year  tour 
of  duty.  During  that  same  period,  he  was  the  Chief  of  Naval 
Training  UC-12B  and  C-131  NATOPS  evaluator. 

Throughout  Mr.  Stewart's  Navy  career  he  was  responsible  for 
increased  training  efficiency  initiatives  including  radical 
downward  manpower  restructuring  of  Navy  Training  Squadrons  in  the 
wake  of  contractor  maintenance  programs  and  significant  increases 
in  the  use  of  simulator  training  time  within  Navy  primary  and 
advanced  propeller  training. 


JOHN  M.  THORN  BIOGRAPHICAL  DATA 


Mr.  Thorn  possesses  twenty-six  years  of  simulation  experience  and 
has  held  a  variety  of  positions  in  all  areas  of  the  simulation/ 
simulator  environment  including  maintenance,  design,  integration, 
and  test. 

Currently,  Mr.  Thorn  is  Manager  of  the  Test  Engineering  Department 


of  McDonnell  Douglas  Training  Systems,  St. 
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Manager  -  Test  Engineering  Section 
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A310/B747  Simulators 

Reflectone 

1981-82 

o 

Project  Engineer  CH-46D 
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Project  Engineer  (R  &  D)  at  Crew 
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Site  Manager  -  Mather  AFB 
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FLOYD  WEAVER  BIOGRAPHICAL  DATA 


Floyd  W.  Weaver  is  presently  the  Lead  Simulator  Test  Pilot  for 
FlightSafety  Services  Corporation's  C-5B  Aircrew  Training  System 
located  at  Altus  Air  Force  Base,  Oklahoma.  Mr.  Weaver's 
association  with  Services  Corporation  began  on  March  31,  1986  when 
he  was  selected  to  perform  acceptance  testing  of  six  new  C-5B  FAA 
Phase  II  Weapons  System  Trainers  and  four  redesigned  Crew  Procedure 
Trainers  built  by  CAE-Montreal  for  United  States  Air  Force  aircrew 
training. 

Mr.  Weaver  graduated  from  Memphis  State  University  at  Memphis, 
Tennessee,  with  a  Bachelor  of  Science  in  Industrial  Psychology.  He 
was  then  commissioned  in  the  United  States  Air  Force  and  selected 
for  Undergraduate  Pilot  Training,  completing  qualifications  as  an 
Air  Force  pilot  in  November,  1960.  Additional  educational  studies 
include  Professional  Military  Education  and  graduate  courses  in 
business  management. 

Mr.  Weaver's  career  in  the  Air  Force  spanned  a  period  of  over 
twenty-six  years.  His  interest  in  simulator  devices  and  aircraft 
testing  began  in  1970,  when  he  accomplished  in-plant  and  on-site 
acceptance  testing  of  the  original  C-5A  Mission  Flight  Simulator. 
During  his  remaining  Air  Force  career,  he  was  the  Military  Airlift 
Command's  Test  Pilot  to  test  and  accept  several  new  C-5  systems, 
including  the  Ground  Proximity  Warning  System,  Triple  Inertial 
Navigation  System,  and. the  Fuel  Savings  Advisory  System.  He  also 
conducted  the  initial  flight  tests  to  verify  the  compatibility  of 
the  C-5  aircraft  to  air  refuel  with  the  KC-10  aircraft,  with 
emphasis  on  structural  as  well  as  operational  requirements. 

As  part  of  his  current  duties  with  Services  Corporation,  Mr.  Weaver 
flies  the  C-5  airplane  on  a  regular  basis  to  compare  the  simulator 
performance  with  the  actual  aircraft  and  to  test  system 
modifications.  Mr.  Weaver  is  an  FAA  Certified  Flight  Instructor 
with  over  11,400  flight  hours  as  a  pilot. 
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F.J.  "CHIP"  WINTER  BIOGRAPHICAL  DATA 


Mr.  Winter  has  technical  and  managerial  experience  with  both  the 
Federal  Government  and  industry.  He  was  employed  as  a  flight  test 
engineer  by  General  Dynamics  in  Fort  Worth,  Texas,  responsible  for 
conducting  tests  on  advanced  aircraft  weapon  systems.  As  a  senior 
systems  design  engineer  for  Northwest  Airlines  in  Minneapolis, 
Minnesota,  he  was  responsible  for  implementing  system  improvements 
to  the  commercial  aircraft  fleet. 

Employed  by  the  Aeronautical  Systems  Division  (ASD)  at  Wright- 
Patterson  AFB  Ohio  since  1964,  he  has  held  various  positions  on  key 
Air  Force  programs.  He  was  assigned  to  the  Air  Force  Flight 
Dynamics  Laboratory,  Flight  Research  and  Test  Branch,  where  he  was 
project  engineer  for  refined  flight  control  and  display- 
investigations.  He  also  managed  and  conducted  the  Federal  Aviation 
Administration  sponsored  Phase  II  microwave  landing  system  research 
simulation. 

Mr.  Winter  is  currently  assigned  to  the  ASD  Training  Systems  SPO  as 
the  Director  of  Acquisition  Management.  He  is  responsible  for  the 
test  and  deployment  planning  and  implementation  for  each  of  several 
strategic,  tactical  and  airlift  programs.  He  has  written  and 
presented  several  articles  and  technical  reports  and  he  has  been  a 
member  of  the  American  Helicopter  Society,  the  Society  of  Flight 
Test  Engineers,  and  the  American  Institute  of  Aeronautics  and 
Astronautics.  He  is  currently  a  member  of  the  Air  Force 
Association  and  the  Interservice  Industry  Training  Systems 
Conference  steering  committee.  He  holds  a  Bachelor  of  Science 
degree  in  Aeronautics  and  a  Master  of  Arts  degree  in  Psychology. 
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APPENDIX  B 

CURRENT  TEST  PROCESS  FLOWCHART 
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DESCRIPTION  Of  THE  CURRENT  TEST  PROCESS  FLOWCHART 


Page  1  of  the  flowchart  shows  the  RFP/Proposal/Contrac*- 
Award  process  where  the  test  requirements  are  first  addressed  and 
agreed  to.  The  system  requirements  are  then  reviewed  and  design 
data  is  requested. 

Page  2  describes  the  preliminary  system  design  and  design 
review  processes  where  specific  requests  for  action  (RFAs)  are 
answered  causing  delays  or  worse. .. rejecting  the  PDR. 

Page  3  describes  the  Critical  Design  Review  (CDR)  and  the 
hardware/ software  design  processes.  Generation  of  test  procedures, 
hardware  drawing  release,  and  software  coding  all  follow  CDR 
approval . 


Page  4  depicts  how  testing  is  begun  with  subsystem  or 
unit  testing  and  then  continued  with  hardware/software  integration. 
Any  problems  encountered  require  redesign  and  rewrite  of  test 
procedures. 

Page  5  shows  the  contractor  verification  test  cycle  and 
results  in  a  statement  of  test  readiness  to  the  Air  Force.  Test 
Readiness  Review  (TRR)  #1  is  conducted  prior  to  a  Computer  Program 
System  Generation  (CPSG)  test  or  "cold  start". 

Page  6  shows  the  cold  start  process  and  the  functional 
check  of  the  device  that  determines  operational  status  of  the 
software  load  prior  to  the  second  TRR.. 

Page  7  describes  the  Government  in-plant  performance  test 
and  operational  evaluation  processes  including  TD  resolution. 

Page  8  shows  TRR  #4  to  determine  if  the  device  is  ready 
to  ship.  The  contractor  on-site  installation  and  verif icaticr, 
tests  are  also  shown  prior  to  on-site  Government  acceptance 
testing . 

Page  9  shows  the  last  test  cycle,  operational  evaluation, 
and  TD  recheck,  prior  to  the  final  acceptance. 
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PROCESS  ANALYSIS 
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Symbols  are  the  language  of  Process  Analysis. 
There  are  nine  symbols,  six  of  which  represent 
different  types  of  tasks.  They  are: 

Operation  -  means  to  create  or  change 
something. 

Transportation  -  denotes  physical  movement. 
Inspection  -  used  when  something  is  checked 
or  verified. 

Delay  -  represents  any  temporary  stoppage  in 
work. 

Decision  -  used  whenever  a  decision  causes 
the  workflow  to  split;  for  example,  into  separate 
*o.k.“  and  “not  o.k.“  paths. 

File  -  used  to  represent  the  filing  of  documents 


There  are  three  others  which  are  useful: 

Document  -  used  for  any  piece  of  paper 
Computer  -  represents  any  computer  or 
computer  system. 

Connector  -  used  to  link  different  parts  of  the 
diagram  whenever  drawing  flow  lines  is 
confusing  or  impractical. 

The  PA  kit  contains  hundreds  of  each  of  these 
symbols.  A  description  of  each  task  or  activity 
in  the  workflow  can  be  written  directly  on  the 
symbol  and  the  symbol  can  be  applied  to  the 
charting  paper. 
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PROCESS  ANALYSIS 


Process  Analysis 


To  construct  the  chart,  the  charting  paper  is 
attached  to  the  wall.  The  team  identifies  the 
different  departments  or  groups  involved  in  the 
process  and  assigns  a  horizontal  section  of  the 
chart  to  each.  Then,  symbols  are  added  to  the 
chart  to  represent  the  beginning  and  ending  of 
the  process.  The  task  remaining  is  to  fill  in  the 
blank  space. 

As  the  team  continually  asks,  "What  happens 
next?"  appropriate  symbols  are  added  with 
descriptions  and  connecting  flow  lines. 


A  good  flowcharting  tip  is  not  to  get  immediately 
tost  in  the  detail  of  what  happens  when  an  error 
occurs. 

The  first  time  though,  the  team  should  note  the 
decision  points  and  assume  everything  is  "o  k.” 
Then,  the  detail  for  the  "not  o  k."  paths  can  be 
added.  The  resulting  chart  will  be  much  more 
readable. 


Process  Analysis 


A  flowchart  for  a  cross  functional  process  might 
look  like  this  when  completed 
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TRAINING  SYSTEM  TYPES 


MAINTENANCE  TRAINER 

-  SPECIFIC  SYSTEM/SUBSYSTEM  TASKS 

PART  TASK  TRAINER  (PTT)  -  AIRCREW  &  MAINTENANCE 

-  DISCRETE  TASKS 


OPERATIONAL  FLIGHT  TRAINER  (OFT) 

-  DYNAMICALLY  SIMULATES  ACTUAL  FLIGHT 
CHARACTERISTICS 

-  LIMITED  MISSION  EXECUTION 


MISSION  TRAINER  (MT) 

-  MISSION  ORIENTED  WARFARE  ENVIRONMENT  FOR  WEAPON 
SYSTEMS  OPERATIONS 


WEAPON  SYSTEM  TRAINER  (WST) 

-  FLIGHT  AND  SYSTEMS  OPERATION 

-  MISSION  REHEARSAL 


APPENDIX  D 

DATA  COLLECTION  QUESTIONNAIRE 
(MILITARY  PROGRAMS) 
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QUESTIONNAIRE 


PURPOSE 

To  gather  and  analyze  performance  parameters  associated  with  formal  customer 
test  acceptance.  The  objective  is  to  streamline  the  test  effort  without 
compromising  the  confidence  in  the  test  program.  The  customer  is  not  to  assume 
any  additional  risk  due  to  streamlining  efforts. 

RESULTS 

Your  participation  will  assure  you  of  obtaining  the  findings  determined  by  the 
CPT  and  any  recommendations  being  proposed. 

I.  Data  will  be  collected  in  four  primary  areas: 

Test  Milestones 
Test  Discrepancies 
Test  Duration 
Test  Documentations 

Note :  Only  data  for  the  development  unit  (first  article)  is  solicited. 

II.  Name  of  program:  _ _ 

1.  Test  Milestones: 

Dates  requested  are  those  dates  met  or  missed  with  respect  to  planned 
or  actual  schedules  at  contract  award.  Comment  on  differences. 

Data  requested  for  the  following  specific  start  and  completion  date  if 
applicable/available . 


Original 

Date  Actual 


Customer  Test  Readiness  Review 
Customer  Coldstart 

Customer  in-plant  First  Article  Test 
Readiness  Review 

Customer  in-plant  First  Article 
Development 

Operational  Evaluation 
Teardown/Pack 
Ship 

Customer  on-site  First  Article  Test 
Readiness  Review 


(&£PI1)  late.  3/13/90 
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Original 

Date  Actual 

Customer  on-site  First  Article  Tests 

Acceptance  _  _ 

Operational  Evaluation  _  _ 

Sell  off  (customer  acceptance)  _  _ 

2.  Test  Duration  (Actual  test  days): 

Plan  Actual  Hours  Days/ 

Days  Days  /Day  Week  Comments 


Customer  cold 
start 


Customer  in-plant 
development 

Customer  in-plant 

operational 

evaluation 


Contractor  on¬ 
site  installation 
and  test 


Customer  on-site 
acceptance 

Customer  on-site 

operational 

evaluation 


3.  Test  Discrepancies: 

Number  of  contractor  discrepancies  open  at  TRR  _ 

Number  of  contractor  discrepancies  at  start  of  tests  _ 

Total  number  of  customer  discrepancies  (first  unit)  _ 

Number  of  customer  discrepancies  resubmitted  _ 

(A  discrepancy  resubmitted  four  times  would  count  as  four) 

Number  of  contractor  discrepancies  from  internal  Dry  Run  _ 

Number  of  customer  discrepancies  post  sell-off  _ 

Number  of  customer  discrepancies  Judged  out-of-scope  _ 

Number  of  customer  discrepancies  against  test  procedures  _ 


(Bffll)  Date:  3/13/90 
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4.  Test  Documents : 


Development  test  procedures. 

Number  of  pages  _ 

Number  of  steps/page  _ 

Number  of  steps  for  major 

subsystems  (l.e.,  radar,  _  _  _  _ 

visual  aero,  poverplant, 
motion,  etc.) 

Test  Matrix 

Total  number  of  inspections  _ 

Total  number  of  analysis  _ 

Total  number  of  demonstrations  _ 

Total  number  of  tests  _ 

5.  Engineering  change  activity 

CDR  through  start  of  customer  tests.  Comment  with  respect  to  impact 
on  the  test  process.  Qualify  if  possible,  i.e.,  ECN ,  SLR's,  etc. 

o  CDR  to  start  of  Hardware/Software  Integration  _ 

o  Hardware/Software  Integration  to  TRR  _ 

o  TRR  to  sell  _ 

Range  the  type  of  change  activity  that  most  impacts  test . 

o  Intemal/Engineering  activity  _ 

o  Changes  to  close  customer  DR’s  _ 

o  Incorporation  of  late  data, 

configuration/performance  change  etc.  _ 

6.  Program  Complexity  Indicators: 

a.  Lines  of  Code  _ 

b.  Number  of  CPU’s  _ _ 

c.  Major  subfunctions  _ 

d.  Other  significant  characteristics 


7  Provide  any  specific  comments  or/and  recommendations  which  would  in 
your  opinion  make  the  formal  Customer  Test  phase  more  cost  effective. 


(Ml)  Data:  3/13/90 
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DATA  COLLECTION  QUESTIONNAIRE 
(COMMERCIAL  AIRLINES) 
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QUESTIONNAIRE 


PURPOSE 

To  gather  and  analyze  performance  parameters  associated  with  formal  customer 
test  acceptance.  The  objective  is  to  streamline  the  test  effort  without 
compromising  the  confidence  in  the  test  program.  The  customer  is  not  to  assume 
any  additional  risk  due  to  streamlining  efforts. 

RESULTS 

Your  participation  will  assure  you  of  obtaining  the  findings  determined  by  the 
CPT  (Critical  Process  Team)  and  any  recommendations  being  proposed. 

I.  Data  will  be  collected  in  fou’"  p”i ■*‘~vy  areas. 

Test  Schedules 
Test  Discrepancies 
Test  Duration 
Test  Documentations 

II.  Name  of  program:  _ 

1.  Test  Schedule: 

Dates  requested  are  those  dates  met  or  missed  with  respect  to  planned 
or  actual  schedules  at  contract  award.  Comment  on  differences. 

Data  requested  for  the  following  specific  start  and  completion  date  if 
applicable/available . 


Original 

Date  Actual 

Customer  in-plant  Engineering  Test  _  _ 

Customer  in-plant  FAA  Evaluation  _  _ 

Teardown/Pack  _  _ 

Ship  _ _ 

Contractor  on-site 

installation  and  test  _  _ 

First  Readiness  Review  _ 

Acceptance  Testing  _  _ 

FAA  Evaluation  _ 

Sell  off  (customer  acceptance)  _ 


(B PCFS2)  Date:  3/13/90 
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2.  Test  Discrepancies: 

Number  of  customer  discrepancies  _ _ 

Number  of  open  customer  discrepancies  post  sell-off  _ 

Number  of  customer  discrepancies  judged  out-of-scope  _ 

Number  of  customer  discrepancies  resubmitted  _ _ 

(A  discrepancy  resubmitted  four  times  would  count  as  four) 

3.  Test  Duration  (Actual  test  days): 


Plan 

Actual 

Hours 

Days/ 

Days 

Days 

./Pa_Y- 

Week 

Comments 

Customer  in-plant 
engineering  tests 

Contractor  in- 
plant  FAA  test 

Contractor  on-site 
installation  and 
test 


Customer  on-site 
tests 

FAA  Evaluation 


4 .  Test  Documents : 

Engineering  test  procedures. 

Number  of  pages  _ 

Number  of  steps/page  _ 

Number  of  steps  for  major 

subsystems  (i.e.,  radar,  _  _  _  _ 

visual  aero,  powerplant, 
motion,  etc.) 

5.  Comment  as  appriopriate  for  those  activities  that  preceeded  the  stant 
of  in-plant  customer  acceptance  tests.  Describe  impact  on  the  test 
process . 


(MCH2)  Cate:  3/13/90 
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PAGE  1 


PROGRAM 

II 

II 

II 

II 

ORGINAl 

TRR 

ACTUAL  DUR 

II 

II 

II 

II 

ORGINAL 

COLD  START 

ACTUAL 

OUR 

II 

II 

II 

II 

ORGINAL 

1st  ART  TRR 

ACTUAL 

DUR 

1 

HUGHES 

PTT 

G8U-15  PTT 

II 

02-Feb-aa 

02-Aug-88  -182 

II 

02 -Aug -88 

03-Aug-88 

-183 

II 

N.A. 

N.A. 

182 

2 

ECC 

PTT 

F-15E  LANTRIN 

II 

29-Mar-88 

29-Apr-88  -31 

II 

02- May- 88 

02-May-88 

-31 

II 

29-Apr-88 

29- Apr- 88 

-31 

3 

A  SO 

CPT 

F-16  AIR  DEFENSE  COCKPIT || 

27-Mar-89 

27- Nov- 89  -245 

II 

28-Mar-89 

28-NOV-89 

-490 

II 

N.A. 

N.A. 

245 

4 

MOAC 

WST 

AV8B  NAUST  DEVICE  2F150 

II 

25-May-89 

29-Mey-89  -4 

II 

22-Nay-89 

22-May-89 

-4 

II 

N.A. 

N.A. 

-4 

5 

ASO 

OFT 

EF-111 

II 

11 -Mar-85 

ll-Mar-85  0 

II 

15-Mar-85 

13-Apr-85 

-29 

II 

15-Apr-85 

15-Apr-85 

0 

6 

LORAL 

UST 

F-15E  UST  E3 

II 

01 -Aug-89 

16- Feb- 90  -199 

II 

19- Jan-90 

12-Mar-90 

-251 

II 

29- Jan-90 

27-Mar-90 

256 

7 

ECC 

MT 

F-16C  ELEC  MAINT  TR 

II 

??? 

???  0 

II 

??? 

??? 

0 

II 

??? 

??? 

0 

3 

ECC 

PTT 

F-16C  LANTRIN 

II 

27-Nov-88 

27- Jan-89  -61 

II 

27- Jan -89 

27- Jan-89 

-61 

II 

27- Jan-89 

27- Jan-89 

•61 

9 

ECC 

MT 

F-16C  PROP  MAINT  TR 

II 

??? 

???  0 

II 

??? 

??? 

0 

II 

??? 

??? 

0 

10 

ECC 

MT 

F-16C  ENG  OPS  MAINT  TR 

II 

??? 

???  0 

II 

??? 

??? 

0 

II 

??? 

??? 

0 

11 

HUGHES 

MT 

F-16  TFE-19  ARMAMENT 

II 

09- Jan-89 

03 -Apr -89  -84 

II 

09-Mar-89 

03 -Apr -89 

-109 

II 

N.A. 

N.A. 

-84 

12 

HUGHES 

MT 

F-15  MSIP  TFE-15 

II 

15-May-88 

01-Aug-88  -78 

II 

01-Aug-88 

01 -Aug-88 

-78 

II 

N.A. 

N.A. 

-78 

13 

HUGHES 

MT 

F-15  MSIP  TFE-16 

II 

16-Apr-88 

01-Sep-88  -138 

II 

01 -Sep-88 

01 -Sep-88 

-138 

II 

N.A. 

N.A 

138 

14 

ASD 

MT 

CH-47  El  SYS 

II 

N.A. 

N.A.  0 

II 

N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

15 

ASO 

MT 

AH- IS  EL  SYS 

II 

N.A. 

N.A.  0 

II 

N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

16 

ASD 

MT 

UH-1N  BASIC 

II 

N.A. 

N.A.  0 

II 

N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

17 

ASO 

UST 

B- IB  SIMULATOR  SYSTEM 

II 

18-Oct-88 

18-Oct-88  0 

II 

24-Oct-88 

lO-Nov-88 

-17 

II 

15-NOV-88 

10-Mar-89 

115 

BENCHMARK 

II 

II 

PROGRAMS  |  | 

II 

II 

1 

START  ENGINEERING  TEST  | 

ORGINAL  ACTUAL  DUR  | 

1 

1 

[  NOT  APPLICABLE 

[  COLD  START 

ORGINAL  ACTUAL 

DUR  | 

1  NOT  applicable 

|  1st  ART  TRR 

|  ORGINAL  ACTUAL 

DUR 

BCAC 

737-300  STD  #1  1 1 

01 -Feb-86 

27- Jan-86 

5  | 

5  | 

5 

2 

USAir 

737-300  #4  Phase  II  || 

lO-Oct-89 

26- Jan-90 

-108  | 

-108  | 

108 

3 

DELTA 

MO 88  #1  PHASE  II  SIMUL'R| | 

18-Sep-87 

06- Jan-88 

-110  | 

-110  | 

no 

4 

NU 

AIRBUS  A320  SIMULATOR  #3|| 

0  | 

0  | 

0 

5 

OELTA 

757  *2  AST  PHASE  II  SIM  | | 

0  | 

0  | 

-1 

6 

UAL 

747-400  | | 

19-Sep-88 

l9-Sep-88 

0  | 

0  I 

0 

7 

FS  AST  C-58  UST  - 1  |  | 

28- Apr -86 

28-Apr-86 

0  j 

27- Jun-86 

27- Jun-86 

0  I 

|  N.A. 

N.A. 

3 

8 

CAE 

NUCLEAR  POUER  PLANT  SIM  || 

0  | 

0  I 

0 

9 

ASO 

HS601  TEST  EQ'T  (SAT)  || 

01 -Mar-90 

01 -Apr-90 

-31  | 

N.A. 

N.A. 

-31  1 

1  01 ‘Mar-90 

07 -Apr- 90 

62 
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PROGRAM 

II 

II 

II 

11 

ORGINAL 

DEVELOPMENT 

ACTUAL  OUR 

II 

II 

II 

II 

ORGINAL 

OP  EVAL 

ACTUAL 

OUR 

II 

II 

II 

II 

ORGINAL 

TEARDOVJN/PACK 

ACTUAL  DU 

1 

HUGHES 

PTT 

GBU-15  PTT 

II 

04-Aug-88 

20-Aug-88 

-198 

II 

22-AUS-88 

02-Sep-8a 

-193 

II 

06- Feb-89 

09- Feb- 89 

2 

ECC 

PTT 

F-15E  LANTRIN 

II 

02-May-88 

02-May-88 

-31 

II 

16-May-88 

16-May- 88 

-31 

II 

03- Jun-88 

03- Jun-86 

z 

3 

ASO 

CP  T 

F-16  AIR  DEFENSE  COCKPIT) | 

05 -Apr -89 

05-Dec-89 

-489 

II 

N.A. 

N.A. 

-245 

II 

??? 

??  / 

4 

MO  AC 

WST 

AV88  NAUST  OEVICE  2F150 

II 

30- Jun-89 

30 -Jun-89 

-4 

II 

N.A. 

N.A. 

-4 

II 

10- Jul-89 

18- Jul -84 

5 

ASO 

OFT 

EF- 111 

II 

30 -Apr -85 

15- Apr-85 

15 

20-Mar-86 

15-Aug-86 

-148 

II 

??? 

6 

LORAL 

WST 

F-15E  WST  E3 

II 

29-Jan-90 

28-Mar-90 

-257 

ii 

OI-Jul-90 

15-Sep-90 

-275 

II 

23- Jul -90 

1 5  - Oc  t  *  9C 

-  7  t 

7 

ECC 

MT 

F-16C  ELEC  MAINT  TR 

II 

??? 

??? 

0 

ii 

??? 

??? 

0 

II 

??? 

3 

ECC 

PTT 

F-16C  LANTRIN 

II 

30- Jan-89 

30- Jan-89 

-61 

ii 

13- Feb- 89 

13- Feb-89 

-61 

II 

25 -Feb- 89 

25-  Feb- 39 

9 

ECC 

MT 

F-16 C  PROP  MAINT  TR 

II 

??? 

??? 

0 

ii 

??? 

??? 

0 

II 

??? 

10 

ECC 

MT 

F-16C  ENG  OPS  MAINT  TR 

II 

??? 

??? 

0 

ii 

??? 

??? 

0 

II 

??? 

“>‘>7 

11 

HUGHES 

NT 

F-16  TFE-19  ARMAMENT 

II 

17- May -89 

17-May- 89 

-84 

ii 

17-May-89 

17-May-89 

-84 

II 

02- Jun-89 

02-Jun-84 

12 

HUGHES 

MT 

F- 15  MS IP  TFE-15 

II 

16-0ct -88 

22-Nov-88 

-115 

ii 

16-Oct-88 

22-Nov-88 

-115 

II 

25-Nov-88 

25-NOV-88 

13 

HUGHES 

MT 

F - 15  MS IP  TFE-16 

II 

Q1-0et-88 

22-Nov-88 

-190 

ii 

OI-Oct-88 

22-Nov-88 

-190 

II 

25-NOV-88 

25 -Nov -88 

:T 

14 

ASO 

MT 

CH-47  EL  SYS 

II 

N.A. 

N.A. 

0 

ii 

1Q-Jun-85 

10- Jun-85 

0 

II 

13- Jun-85 

13- Jun*85 

13 

ASO 

MT 

AH- IS  EL  SYS 

II 

N.A. 

N.A. 

0 

ii 

30- Jan- 86 

30- Jan-86 

0 

II 

20- Feb-86 

20- Feb- 86 

’6 

ASO 

MT 

UH-1N  BASIC 

II 

N.A. 

N.A. 

0 

ii 

29-Oct-86 

29-OCI-86 

0 

II 

05-Nov-86 

05-NOV-86 

17 

ASO 

WST 

B- 18  SIMULATOR  SYSTEM 

II 

02-0ec-88 

20-Mar-89 

-108 

ii 

14-Mar-89 

12- Jan-90 

-304 

II 

N.A. 

II 

II 

PROGRAM  |  | 

II 

II 

ENGINEERING  TEST 

ORGINAL  ACTUAL 

DUR 

II 

II 

II 

Ii 

II 

FAA  EVALUATION 

ORGINAL  ACTUAL 

OUR 

II 

II 

II 

II 

II 

ORGINAL 

TEAROOWN/ -ALv 

ACTUAL  : 

i 

BCAC 

737-300  STO  #1  | | 

04-Apr-86 

14 -Mar-86 

26 

II 

NONE 

NONE 

5 

II 

25 -Apr-86 

31-Mar-?'- 

2 

USAir 

737-300  *4  Phase  II  || 

IO-Nov-89 

24- Jan-90 

183 

II 

N/A 

-108 

II 

3 

DELTA 

MO 88  #1  PHASE  II  SIMUL'R| | 

18-Sep-87 

06- Jan- 88 

220 

II 

N/A 

N/A 

-110 

II 

09-Oct-87 

11 -Apr-38  ;o 

u 

NU 

AIRBUS  A320  SIMULATOR  #3|| 

N/A 

N/A 

0 

II 

N/A 

N/A 

0 

II 

15 -Oct -89 

02-NOV-S9  -’ 

5 

OELTA 

757  *2  AST  PHASE  II  SIM  1 1 

NONE 

NONE 

0 

II 

NONE 

NONE 

0 

II 

7?? 

30-Mar  -  35  *** 

6 

CAE 

UAL  747-400  | | 

19-Sep-8S 

19-Sep-88 

0 

II 

N/A 

0 

II 

28-Nov-88 

28-NOV-88 

7 

FS 

AST  C-5B  UST-1  |  | 

24* Jun-86 

24- jun-86 

0 

II 

24- Jun-86 

24- Jun-86 

0 

II 

1 1 -Jul -86 

1 1  - Jul -  86 

8 

CAE 

NUCLEAR  POWER  PLANT  SIM  || 

22- Jan-88 

23-Oct-87 

91 

II 

N/A 

0 

1 1 

07-Apr*88 

21 -Mav- 88 

9 

HS601  TEST  EQUIPMENT  |  | 

01 -May-89 

01 -May-89 

-31 

II 

01 -Feb- 90 

01 -Feb-90 

-31 

II 

15-Mar-90 

22- Apr -4.  r 
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OCT 

26, 

1990 

PROGRAM 

II 

II 

II 

II 

II 

ORGiNAL 

SHIP 

ACTUAL 

DUR 

II 

II 

II 

II 

ORGINAL 

ON-SITE  TRR 

ACTUAL  DUR 

II 

II 

II 

II 

II 

ON-SITE  ACCEPTANCE 

ORGINAL  ACTUAL  DUR 

1 

PTT 

G8U-15  PTT 

II 

06- Feb- 89 

09- Feb-89 

-185 

II 

05- Jun-89 

05- Jun-89 

-182 

II 

06- Jun-89 

16*  Jun-89 

-192 

2 

PTT 

F- 156  LANTRIN 

II 

03-Jun-88 

03- Jun-88 

-31 

II 

19- Jun-88 

19- Jun-88 

-31 

II 

20- Jun-88 

20- Jun-88 

-31 

3 

CPT 

F-16  AIR  OEFENSE  COCKPITj  j 

24-JUI-89 

25 -Jan- 90 

-430 

II 

05- Jun-89 

Q5-Feb-90 

-490 

II 

12- Jan-90 

08- Feb-90 

-in 

4 

VST 

AV88  NAUST  DEVICE  2F150 

II 

24-Jul-89 

24-Jut-89 

-4 

II 

N.A. 

N.A. 

-4 

II 

06-Nov-89 

06- Nov* 89 

4 

5 

OFT 

EF-111 

II 

??? 

??? 

0 

II 

01 -Nov-86 

01 -Nov-86 

0 

II 

13-Nov-86 

15-0ec-86 

32 

6 

UST 

F-15E  VST  E3 

II 

15-Aug-90 

30-0ct-90 

-275 

II 

15-0ct-90 

02- Jan-91 

-278 

II 

19-NOV-90 

07- Feb-91 

-279 

7 

MT 

F-16C  ELEC  MAINT  TR 

II 

??? 

??? 

0 

II 

??? 

??? 

0 

II 

c 

8 

PTT 

F-16C  LANTRIN 

II 

25-Feb-89 

25 -Feb- 89 

•61 

II 

04 -Mar -89 

04-Mar-89 

-61 

II 

06-Mar-89 

06-Mar-89 

-61 

9 

MT 

F-16C  PROP  MAINT  TR 

II 

??? 

??? 

0 

II 

??? 

??? 

0 

II 

??? 

0 

10 

MT 

F-16 C  ENG  OPS  MAINT  TR 

II 

??*? 

??? 

0 

II 

??? 

??? 

0 

II 

??? 

■)')-> 

0 

11 

MT 

F-16  TFE-19  ARMAMENT 

II 

02-Jun-89 

02- Jun-89 

-84 

il 

N.A. 

N.A. 

-84 

II 

N.A. 

N.A. 

•84 

12 

MT 

F- 15  MS  IP  TFE-15 

II 

25 -Nov- 88 

25-NOV-88 

-78 

II 

28-Nov-88 

28-NOV-88 

-78 

II 

22-Dec-88 

22-Dec-88 

-78 

'3 

MT 

F- 15  MSI P  TFE-16 

II 

25-Nov-88 

25-NOV-88 

-138 

II 

29-Nov-88 

29-Nov-88 

-138 

II 

13-Dec-88 

13-0ec-88 

-138 

K 

MT 

CH-47  EL  STS 

II 

14- Jun-85 

14-Jun-85 

0 

II 

N.A. 

N.A. 

0 

II 

20- Jun-85 

16- Feb-90 

15 

MT 

AH-1S  EL  STS 

II 

21 -Feb- 86 

21  -  Feb-86 

0 

II 

N.A. 

N.A. 

0 

II 

24- Feb-86 

24- Feb- 86 

0 

16 

MT 

UH-1N  BASIC 

II 

15-NOV-86 

15-Nov-86 

0 

II 

N.A. 

N.A. 

0 

II 

19-NOV-86 

19-Nov-86 

0 

17 

UST 

3-18  SIMULATOR  SYSTEM 

II 

N.A. 

0 

II 

N.A. 

0 

II 

N.A. 

0 

II 

II 

PROGRAM  | ] 

II 

II 

ORGINAL 

SHIP 

ACTUAL 

II 

II 

DUR  |  | 

II 

i  i 

FIRST  READINESS  REVIEU 

ON-SITE  TRR 

ORGINAL  ACTUAL  DUR 

II 

II 

11 

II 

II 

acceptance  testing 

on-site  acceptance 

ORGINAL  ACTUAL  DUR 

1 

737-300  STD  *1  || 

02 -May- 86 

07-Apr-86 

30  || 

26-May-86 

12-May-86 

19 

II 

13- Jun-86 

18- Jun-86 

0 

2 

737-300  #4  Phase  II  | | 

1 1 -Dec -89 

27- Jan-90 

-155  || 

N.A. 

N.A. 

-108 

II 

02- Feb-90 

27-Msr-90 

161 

3 

MO 88  #1  PHASE  II  SIMUL'R | | 

16-0ct -87 

16-Apr-88 

-293  || 

-110 

II 

18-Dec-87 

15 -May-88 

259 

AIRBUS  A320  SIMULATOR  #3  I | 

IS-Oct-89 

08-Nov-89 

-24  || 

N.A. 

N.A. 

0 

II 

10-Mar -90 

13-Mar  *90 

-3 

5 

757  «  AST  PHASE  II  SIM  | | 

04 -Apr -89 

. II 

0 

1  1 

-  n  nr 

-J  V/  vWM  .  u  1 

04  Jul  89 

6 

UAL  747-400  | | 

29-NOV-88 

29-NOV-B8 

o  II 

N.A. 

N.A. 

0 

II 

23- Jan-89 

23- Jan- 89 

0 

1 

AST  C - 58  UST  - 1  |  | 

ll-Jul-86 

ll-Jul-86 

o  II 

26- Jul -86 

26- Jul -86 

0 

II 

03 -Oct -86 

0 1  - Oc  t -  86 

2 

8 

NUCLEAR  POWER  PLANT  SIM  || 

14-May-88 

16- Jun-88 

-33  || 

N.A. 

N.A. 

0 

II 

14- Aug-88 

22- Sep-88 

39 

9 

NS601  TEST  EQUIPMENT  || 

16-Mar-90 

23 -Apr -90 

-69  || 

20 -Mar -90 

01 -May-90 

-73 

II 

27-Mar -90 

08 -May -90 

-73 

REVISED 
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PROGRAM 

||  ON-SITE  OPS  EVAL 

|  orginal  actual 

1 

II 

OUR 

II 

II 

II 

II 

ON-SITE  SELL-OFF 

ORGINAL  ACTUAL 

DUR 

1 

HUGHES 

PTT 

GBU-15  PTT 

|  18-Sep-89 

20- Sep- 89 

-184 

II 

21-Sep-89 

21 -Sep- 89 

-182 

2 

ECC 

PTT 

F-15E  L ANTRIM 

|  27- Jin-88 

27-Jl*i-88 

-31 

II 

13-jui-aa 

13-JUI-88 

-31 

3 

A  SO 

CPT 

F-16  AIR  DEFENSE  COCKILPIT 

|  N.A. 

N.A. 

-245 

II 

06- Jul-69 

08- Feb- 90 

-462 

4 

MOAC 

UST 

AV8B  NAWST  DEVICE  2F150 

|  IS-Nov-89 

IS-Nov-89 

-4 

II 

IS-Nov-89 

15-0ee-89 

-34 

5 

ASD 

OFT 

EF-111 

|  27-Oct-86 

09- Nov- 86 

-13 

II 

31-Jan-86 

20-Feb-37 

-385 

6 

LORAL 

UST 

F-15E  UST  E3 

|  Q1 -Nov-90 

16- Jan-91 

-275 

II 

19-NOV-90 

07- Feb-91 

-279 

7 

ECC 

MT 

F-16C  ELEC  MAINT  TR 

1  ??? 

??  ? 

0 

|  1 

??? 

0 

a 

ECC 

PTT 

F- 16C  LANTRIN 

|  13-Mar-89 

13 -Mar -89 

-61 

II 

25-Mar-89 

25-Mar-89 

-61 

9 

ECC 

MT 

F-16C  PROP  MAINT  TR 

|  ??? 

??? 

0 

II 

??? 

??? 

0 

10 

ECC 

MT 

F-16C  ENG  OPS  MAINT  TR 

1  ??? 

??? 

0 

II 

??? 

??? 

0 

11 

HUGHES 

MT 

F-16  TFE-19  ARMAMENT 

|  N.A. 

N.A. 

-84 

II 

19-May-89 

01 -Jun-89 

-97 

12 

HUGHES 

MT 

F- 15  MS  IP  TFE-15 

|  22  Dec -88 

22-Dec-88 

•7B 

II 

22-0ec-88 

22-Dec-88 

-78 

13 

HUGHES 

MT 

F- 15  MS  IP  TFE-16 

|  13-Dec-88 

13-Dec-88 

-138 

II 

13-Dec-88 

13-Dec-88 

-138 

14 

A  SO 

MT 

CH-47  EL  STS 

|  N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

15 

ASO 

MT 

AH- IS  EL  SYS 

|  N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

16 

ASD 

MT 

UH-1N  BASIC 

|  N.A. 

N.A. 

0 

II 

N.A. 

N.A. 

0 

1? 

ASD 

UST 

B-  IB  SIMULATOR  SYSTEM 

|  26-Apr-89 

12- Jun-90 

-412 

II 

30- Apr -89 

28- Jun-90 

-424 

PROGRAM 

II 

II 

II 

II 

II 

FAA  EVALUATION 

ORGINAL  ACTUAL 

DUR 

II 

11 

11 

II 

II 

CUSTOMER 

3EL 

ORGINAL 

ACCEPTANCE 

..-OFF 

ACTUAL 

DUR 

1 

SCAC 

737-300  STD  #1 

II 

20- Jun'86 

24- Jun-86 

1 

II 

23- Jun-86 

25- Jun-86 

3 

2 

USA  \  r 

737-300  #4  Phase  11 

II 

N.A. 

IO-May-90 

|  j 

02- Feb- 90 

28-Mar-90 

162 

3 

DELTA 

MOSS  #1  PHASE  II  SIMUL'R 

i  i 

cc ■ Jan- C8 

v  Jul ■ 88 

-271 

II 

?9-  ia-88 

01 - Jul 

**i *  f 

4 

NU 

AIRBUS  A320  SIMULATOR  *3 

n 

15-Mar-90 

22-Mar-90 

-7 

II 

0 

5 

OELTA 

757  02  AST  PHASE  11  SIM 

ii 

01  - Jul '89 

10- Jul -89 

-9 

II 

01  - Jul -89 

1 1  - Jul -89 

-10 

6 

CAE 

UAL  747-400 

ii 

26- Jan-89 

26 • Jan- 89 

0 

II 

27- Jan-89 

27- Jan-89 

0 

7 

FS 

AST  C-5B  UST - 1 

ii 

1 7  - Oc 1  -  86 

1 7-Oct -86 

0 

II 

03-Oct -86 

03-Oct -86 

0 

8 

CAE 

NUCLEAR  POWER  PLANT  SIM 

ii 

N.A. 

N.A. 

0 

II 

14-Aug-88 

22-Sep-8S 

-39 

Q 

HS601  TEST  EQUIPMENT 

u 

31 -Mar-90 

tN-UORK 

Ml 

31  -Mar-90 

12- May-90 

-73 
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REVISED 


OCT 

2f , 

1990 

PROGRAM 

II 

II 

II  PL 
II 

II 

II 

COLD  START  | | 

ACT  HRS  DAYS | J  PL 
PER  PER  || 

DAY  ux  ; | 

IN-PLANT  || 

OEVEL  | | 

ACT  HRS  DAYS || 
PER  PER  || 
DAY  UK  || 

PL 

IN-PLANT 

OPS  EVAL 

ACT  HRS  DAYS 

PER  PER 

OAY  UK 

1 

PTT 

G8U-15  PTT 

II  5 

2 

8 

5 

II 

18 

29 

12 

5 

II 

30 

15 

8 

5 

2 

PTT 

F-15E  UNTRIM 

II  2 

2 

10 

5 

II 

18 

18 

10 

5 

II 

30 

30 

10 

5 

3 

CPT 

F-16  AIR  DEFENSE  COCKP!T| |  7 

? 

? 

? 

II 

8 

5 

7 

7 

||  N/A 

N/A 

N/A 

N/A 

4 

VST 

AV88  NAUST  DEVICE  2F150 

II  3 

3 

10 

6 

II 

38 

54 

10 

6 

II 

1 

1 

a 

5 

5 

OFT 

EF-111 

II 

II 

II 

6 

US7 

F - 156  UST  E3 

1 1  12 

10 

16 

6 

II 

156 

12 

5 

II 

14 

7 

Ml 

F-16C  ELEC  MAINT  TR 

1  1 

7 

7 

7 

II 

32 

32 

10 

5 

II 

7 

7 

7 

7 

3 

PTT 

F-16C  LANTR1N 

1 1  2 

2 

10 

5 

II 

18 

18 

10 

5 

II 

20 

20 

10 

5 

9 

MT 

P-16C  PROP  MAINT  TR 

1  1  ? 

7 

7 

7 

i  1 

35 

55 

10 

5 

II 

7 

7 

? 

7 

10 

MT 

F-16C  ENG  OPS  MAINT  TR 

1  1  ? 

? 

? 

7 

ii 

27 

27 

10 

5 

II 

7 

7 

7 

? 

11 

MT 

F-16  TFE-19  ARMAMENT 

1 1  1 

1 

8 

5 

ii 

32 

32 

12 

6 

II  N/A 

N/A 

N/A 

N/A 

12 

MT 

F - 15  MS  IP  TFE-15 

II  2 

2 

8 

5 

ii 

33 

33 

12 

6 

||N/A 

N/A 

N/A 

N/A 

13 

MT 

F - 15  MS  IP  TFE-16 

I  1  1 

1 

8 

5 

n 

30 

52 

12 

5 

||  N/A 

N/A 

N/A 

N/A 

'U 

MT 

CH-47  EL  STS 

1  I  N/A 

M/A 

N/A 

N/A 

M  N/A 

N/A 

N/A 

N/A 

II 

10 

10 

7 

? 

’5 

MT 

AH- IS  EL  SYS 

II  M/A 

M/A 

M/A 

M/A 

i  1  N/A 

N/A 

N/A 

N/A 

II 

3 

3 

7 

7 

'6 

MT 

'JH-IN  BASIC 

1  ! 

M/A 

N/A 

M/A 

||N/A 

N/A 

N/A 

N/A 

II 

3 

3 

7 

? 

*  7 

JS' 

3- ’8  SIMULATOR  SYSTEM 

1  1  21 

21 

16 

7 

II 

50 

52 

12 

5 

II 

14 

24 

7 

II 

11 

IN-PLANT 

II 

IN-PLANT 

II 

COLO  START 

1  i 

OEVEL 

11 

OPS 

EVAL 

PROGRAM  J | 

PL 

ACT 

HRS 

DAYS 

1 !  pl 

ACT 

HRS 

OATS 

1  1  PL 

ACT 

HRS 

DATS 

ii 

PER 

PER 

II 

PER 

PER 

II 

PER 

PER 

ii 

OAY 

UK 

il 

DAY 

UK 

II 

DAY 

UK 

- 

737-300  STD  *1  | | 

N/A 

N/A 

N/A 

N/A 

1  1  AS 

35 

8 

5 

||N/A 

N/A 

N/A 

N/A 

? 

737-300  #4  Phase  II  | | 

1  1  20 

30 

10 

5 

1  |N/A 

N/A 

N/A 

N  'A 

3 

MO 88  *1  PHASE  1 1  SIMUL ' R |  | 

1  1  30 

80 

12 

7 

II  s 

10 

12 

7 

- 

AIRBUS  A320  SIMULATOR  «3 I | 

i  1  0 

0 

II  o 

0 

5 

*57  #2  AST  PHASE  II  SIM  || 

1  |N/A 

N/A 

N/A 

N/A 

1  |!/U 

I/U 

I/U 

I/U 

5 

UAL  747-400  || 

II  39 

39 

12 

7 

||N/A 

N/A 

N/A 

N/A 

7 

as*  :- 58  wst-  i  II 

N/A 

N/A 

N/A 

N/A 

II 

II 

3 

NUCLEAR  pouer  plant  SIM  || 

1  1  54 

135 

8 

5 

1  |R/A 

N/A 

N/A 

N/A 

5 

HS601  TEST  EQUIPMENT  || 

N/A 

||  198 

220 

8 

5 

II  8 

15 

8 

5 

REVISED 

OCT  26, 

PAGE  6 

1990 

PROGRAM 

II 

II 

II 

II 

II 

PL 

ON-SITE 

INSTALL/TEST 

ACT  HRS  DAYS 

PER  PER 

DAY  UK 

1! 

I  |>*i 

II 

II 

ON-SITE 

ACCEPT 

ACT  HRS  DAYS 

PER  PER 

DAY  UK 

1 

|  PL 

1 

ON-SITE 

OPS  EVAL 

ACT  HRS 

PER 

DAY 

DAYS 

PER 

UK 

1 

PTT 

G8U-  i  PTT 

II 

5 

3 

12 

5 

II  14 

11 

10 

5 

1  30 

30 

8 

5 

2 

PTT 

F-15E  LANTRIN 

II 

5 

5 

8 

5 

II  5 

5 

a 

5 

1  10 

10 

8 

5 

3 

CP  T 

F - 16  AIR  DEFENSE  COCKP I T  |  | 

5 

3 

7 

7 

II  5 

3 

7 

7 

[  N/A 

N/A 

N/A 

N/A 

4 

UST 

4V88  NAUST  DEVICE  2F150 

II 

60 

104 

16 

6 

II  « 

42 

12 

6 

1  ’ 

9 

8 

5 

s 

OFT 

Ef-111 

II 

II 

1 

6 

WST 

F-15E  UST  E3 

II 

60 

II  21 

1  U 

7 

MT 

F-16C  ELEC  MAINE  TR 

II 

N/A 

7 

7 

7 

1  lN/A 

|  N .  A 

8 

PTT 

F-T6C  LANTRJN 

II 

4 

4 

10 

5 

II  i 

3 

10 

5 

1  13 

13 

10 

5 

9 

MT 

F-16C  PROP  MAINT  TR 

II 

M/A 

7 

7 

7 

||N/A 

|N.A 

10 

MT 

F-16C  ENG  OPS  MAINT  TR 

II 

? 

7 

7 

7 

II  ’ 

7 

7 

7 

1  2 

7 

7 

? 

n 

MT 

F - 16  TFE-19  ARMAMENT 

II 

M/A 

N/A 

M/A 

N/A 

|  jN/A 

N/A 

N/A  N/A 

|N/A 

N/A 

M/A 

N/A 

12 

MT 

F*  15  MS I P  TFE-15 

II 

1 

1 

12 

5 

II  26 

26 

12 

* 

|N/A 

N/A 

N/A 

N/A 

13 

MT 

F - 15  MSIP  TFE-16 

II 

1 

1 

12 

5 

II  15 

15 

12 

5 

|N/A 

N/A 

N/A 

N/A 

14 

MT 

CH-47  EL  STS 

II 

2 

2 

7 

7 

1 1  4 

4 

7 

? 

|N/A 

N/A 

N/A 

N/A 

15 

MT 

AN-1S  EL  SYS 

II 

1 

1 

7 

7 

1 1  3 

3 

7 

7 

|N/A 

N/A 

N/A 

N/A 

16 

MT 

UN- IN  BASIC 

II 

2 

2 

7 

7 

1!  2 

2 

7 

7 

|N/A 

N/A 

N/A 

N/A 

17 

UST 

3- 13  SIMULATOR  system 

II 

14 

0 

12 

7 

II  20 

0 

10 

5 

1  2 

'6 

7 

11 

11 

PROGRAM  |  | 

II 

II 

CONTRACTOR  ON-  | (CUSTOMER  ON¬ 
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INITIAL  DATA  OBSERVATIONS: 

Notes  recorded  during  analysis 
of  the  data  charts 
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A .  Milestones  -  Met  or  delayed : 

1.  Military  systems  are  always  late  getting  to  test.  Commercial 
systems  are  usually  on  time  getting  to  test.  Why? 

la.  Nothing  can  be  determined  from  available  data  -  the 
following  is  supposition. 

Bad  Data 

Military  is  1st  article 
Lack  of  systems  engineering 
ECP  activity  (?) 

Lack  of/Poor  Management  control . 

2.  Once  test  starts,  it  usually  runs  on  time  or  the  magnitude  of 
delay  is  held  constant.  In  some  cases,  system  delays  are 
reduced  in  test,  in  no  case  does  test  time  increase  more  than 
ten  (10)  percent  beyond  the  time  programmed.  Why  is  this 
true? 


2a.  Statement  in  general  is  validated  by  data!  Caused  by 
work  around  or  padded  TPS  schedules. 

3.  Is  test  time  planned  too  long?  Is  test  planning  pessimistic? 

3a.  The  ratio  of  military  to  commercial  correlates  to  and 
supports  the  fact  that  military  schedules  are  realistic. 
The  fact  that  the  military  schedule  is  80  days  average 
versus  60  days  for  commercial  is  answered  because  of  the 
complexity  of  device. 

Military  has  tactics. 

Waste:  Multiple  TRRs,  cold  starts,  TD  clearance. 

4.  Is  the  start  of  test  planned  too  early? 

4a.  Indicators  are  that  the  planned  start  of  test  is  O.K. 
and  is  realistic  in  terms  of  RFT.  The  problem  is  that 
the  test  starts  before  "we're  ready  to  test." 

Waste:  Excessive  resources  applied  to  meet  schedule. 

5.  Ops/FAA  eval  appears  to  run  on  time.  Why  are  these  scheduled 
goals  met? 

5a.  FAA  &  Crew  availability:  Crews  are  available  for  finite 
time  and  PS  Eval/FAA  Eval  are  a  fixed  scheduled  event: 
a  well  defined  task/time  span  based  on  historical  data 
and  device  maturity. 


G-2 


i 


Why  do  some  programs  recover  schedule  during  testing? 

6a-  The  schedule  recover  is  due  to  sloppy  (excessive) 
Teardown/ Pack/ Ship  duration.  Data  also  indicates  that 
on  site  there  is  extended  work  week/hours  per  day,  added 
resources,  and  added  management  attention. 

Why  is  the  maximum  slip  in  testing  only  five  to  ten  percent? 

7a.  The  slip  of  only  5  -  10%  (additional)  is  indicative  of 

—  Workaround  Plans 

Well  defined  Scope  of  Activity 

Pressure. 

Appears  to  finish  on  schedule  because  - 

—  Maturity  of  Device 

Fewer  Problems  to  Identify 

Reduced  Scope  of  Test  -  ATP,  not  DTP. 

How  did  AIRBUS  and  USAir  737  gain  schedule  time? 

9a.  Data  provides  some  indicators.  AIRBUS  drop  shipped  24 
days  early.  Made  up  time  in  T/P/S.  USAir  had  30  days 
for  T/P/S  and  used  3.  The  schedule  gain  is  not  a 
function  of  Test. 

The  amount  of  Air  Force  Test  Team  preparation  and  experience 
seems  to  be  adequate  since  the  test  schedule  does  not  seem  to 
be  impacted.  Does  preparation  and  experience  impact  on  the 
duration  of  daily  test  time  and  the  number  of  days  per  week 
require  for  test? 

10a.  No  evidence  to  indicate  that  the  qualification  of  AF 
Acceptance  team  impact  test  schedule.  The  quality  oi 
test  personnel  remain  an  important  factor  to  hold 
schedule. 

An  observation  was  made  during  discussions  on  Question 
10  -  There  is  correlation  between  Military  and 

Commercial  Daily  test  time  10  to  12  hours/day,  5  days/ 
week. 


I9S£  PjgC.C.ep.ancigg: 

Why  did  GBU-15  have  so  few  TDs? 

la.  GBU-15  had  3  dry  runs  of  ATP  (including  3rd-  time  with 
QA) .  GBU  was  small  device  -  only  400  pages  in  ATP.  it 


G-3 


must  be  noted  that  this  did  not  result  in  schedule 
reduction  -  Plan  =  60  days/ Actuals  =  70  days. 

2.  Why  do  the  quantity  of  TDs  appear  to  correlate  with  the 
complexity  of  the  training  device  —  but  not  always? 

2a.  F15  LANTIRN  and  EF  III  indicators  do  not  conform  to  TD/' 

complexity  curve.  However,  it  must  be  noted  that  both 
devices  are  prototype  devices  (20  TDs  per  day  is  average 
in  both  Military  and  Commercial) . 

3.  Does  the  quantity  of  TDs  correlate  with  test  duration? 

3a.  Plots  of  days  in  Test  versus  TD  Count  indicates  that  the 
number  of  TDs  reasonably  correlates  to  days  in  Test  -- 
20  TD  per  day  average. 

4.  Why  are  the  number  of  C-5B  re-submits  and  post  sell-off  TDs 
higher  than  the  other  systems? 

4a.  Analysis  of  re-submits  indicates  that  bulk  of  re-submits 
were  in  2  major  subsystems.  Inexperienced  engineers 
assign  to  both  systems. 

The  higher  number  of  post  sell-off  TDs  for  C5  was  the 
result  of  the  sell-off  plan.  The  base-line  unit  was 
unit  #6,  and  design  changes  accumulated  through  all  six 
devices. 

5.  Do  re-submits  become  an  issue  during  the  test  phase?  Why  do 
re-submits  appear  not  to  affect  the  test  schedule? 

5a.  Re-submits  do  not  become  an  issue.  Re-submits  do  not 
affect  schedule  because  there  is  parallel  effort  in  TD 
correction  plus  small  number  of  re-submits. 

Waste:  Lack  of  complete  or  incomplete  engineering 

corrective  action. 

6.  How/Why  do  90%  of  the  TDs  get  resolved  during  the  testing 
period?  Most  TDs  do  not  appear  to  require  major  redesigns,  so 
why  were  these  relatively  minor  TDs  not  caught  and  corrected 
by  the  contractor  prior  to  test? 

6a.  Large  parallel  effort  (off  shift) ,  2nd  unit  available 
(more  asset).  (Opinion)  Design  blindness; 

Interpretation  of  data  or  wrong  data. 

7.  The  customer  seems  willing  to  accept  the  trainer  with  greater 
than  10%  TDs  remaining  at  sell-off.  How  does  this  correlate 
with  the  test  schedule  and  why  will  the  customer  do  this?  How 
long  do  these  post  sell-off  TDs  remain? 

7a.  Customers  desire/need  for  device  forces  customer  to 
start  using  device  with  large  number  of  open  TDs  at 
sell-off  —  TDs  are  minor  in  nature:  TDs  do  not  cause 
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negative  training:  usually  agreement  with  contractor  as 
to  correcting  discrepancies. 

Part  2.  Test  schedule  does  not  allow  time  to  clean  up 
TDs  and  still  meet  RFT  date.  The  customer  needs 
training  device. 

Part  3.  No  data  available. 

8.  Why  do  out-of-scope  TDs  not  affect  the  test  schedule? 

8a.  Out-of-Scope  TDs  do  not  affect  test  schedule  because  the 
TDs  are  Out-of-Scope.  There  are  methods  to  contest 
Out-of-Scope  TDs  without  hampering  program.  On  the 
other  hand,  "Got  to  Have"  will  affect  program. 

9.  Is  there  a  correlation  between  the  number  of  TDs  and  the  start 
of  the  test  schedule? 

9a.  No  correlation  exists  between  Test  Schedule  and  TD 
count . 

10.  How  many  TDs  were  open  at  TRR?  What  is  the  level  of  quality 
internal  to  the  contractor  prior  to  TRR?  How  many  total  TDs 
are  the  result  of  the  contractor  validation?  Are  there 
undocumented  TDs  discovered/known  by  the  contractor  which  are 
unresolved  prior  to  beginning  test? 

10a.  No  definitized  answers  from  respondents. 


C.  Days  in  Test: 

1.  Why  do  the  number  of  planned  test  days  correlate  well  with  the 
actual  test  days? 

la.  See  answer  A-2  and  A-7. 

2.  What  is  the  percent  of  acceptance  test  time  to  total  program 
schedule?  (Total  ratio  of  test  days  to  total  program.) 

2a.  This  data  will  be  provided  to  Bill. 

3.  Why  do  breaks  in  the  test  schedule  for  redesign  activity  not 
seem  to  significantly  affect  days  in  test? 

3a.  Minimal  impact  is  felt  because  of  extended  work  week, 
parallel  effort  on  2nd  device,  desire  to  continue  test, 
addition  resources. 

4.  What  kind  of  workarounds  are  used  to  maintain  test  schedules? 
Do  these  include  Software  Support  Centers,  2nd  Article  tests, 
daily  work  time/days/weekends,  etc.?  How  are  overall 
scheduled  maintained  without  an  order  of  magnitude  increase  in 
test  days?  Observations  are  that  planned  and  actual  days  in 
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test  correlate  well.  Complexity  does  not  seem  to  affect  the 
planned/actual  test  schedule  ratio  —  Why? 

4a.  Part  1  —  Workarounds  used  include  those  cited  in  C3  and 
defer  test  to  site. 

Part  2  —  No  data  required. 

Part  3  —  Complexity  does  not  affect  planned/actual  est 
schedule;  up-front  planning  takes  into  account 
complexity. 


D.  Test  Documents: 

1.  Why  is  the  B-1B  ATP  so  large?  Does  this  indicate  complexity 
or  is  it  an  error?  Why  are  there  18  test  steps  per  page.-1 
This  data  is  suspect. 

la.  Chip  reports  that  data  as  provided  on  B-1B  is  correct. 
Waste:  Excessive  Documentation  (Comment) . 

2.  What  makes  the  BGU-15  only  1.3  steps  per  page? 

2a.  GBU-15  has  2.5  steps/page.  This  document  contains 
numerous  illustrations/graphs. 

3.  What  is  the  correlation  between  the  number  of  ATP  pages,  TDs , 
and  the  length  of  the  test  schedule? 

3a.  No  correlation  exists  between  ATP  and  Test  Schedule. 

4.  Why  does  the  Test  Matrix  categories  not  correlate  between  the 
various  programs? 

4a.  Analysis  is  generally  a  small  number  Test  and  Demo  is  a 
matter  of  interpretation. 

Inspection  -  30  to  40%. 

There  is  a  correlation,  though  basically  an  opinion  not 
totally  supported  by  data. 

5.  To  what  level  were  the  ATPs  written  (i.e.,  qualified  operator, 
inexperienced  person,  etc.)? 

5a.  There  is  no  indication  from  available  data  concerning 
ATP  Level.  CPT  opinion  is  that  the  documents  were 
written  for  semi-qualified  personnel. 

Waste:  Opinion  is  that  document  is  written  at  too  low 

a  level. 
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How  much  FAM  training  does  the  customer  require  to  accomplish 
the  ATPs? 


6a.  Customer  training  is  insignificant  (approx.  1  week) . 

7 .  Why  is  the  Demo  category  of  the  Test  Matrix  percentage  larger 
than  expected? 

7a.  See  D4 . 

8.  Why  can't  we  increase  the  percentage  of  the  Inspection  and  the 
Analysis  categories  of  the  Specification  Verification  Matrix? 
What  is  wrong  with  the  Test  category  being  reduced  to  zero 
percentage?  What  minimum  level  of  the  Test  category  is 
acceptable? 

8a.  In  opinion  of  CPT  members  -  WE  CANJ J 
Opinions  - 

Test  Category  -  Zero  (0) 

Mission  Test  Plan  (MTP)  at  Functional  Level 
Waste:  Duplication  of  contractors  test. 


